# Hybrid Relation Guided Set Matching for Few-shot Action Recognition

arXiv: https://arxiv.org/abs/2204.13423 · HTML: https://mlsanity.com/paper/2204.13423/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 81 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [Part-aware Prototype Network for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2007.06309/) (2020)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)

## Built on by (most cited first, 30 of 50)

- [CLIP-guided Prototype Modulating for Few-shot Action Recognition](https://mlsanity.com/paper/2303.02982/) (2023)
- [MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition](https://mlsanity.com/paper/2304.00946/) (2023)
- [M3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition](https://mlsanity.com/paper/2308.03063/) (2023)
- [MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition](https://mlsanity.com/paper/2405.02077/) (2024)
- [Boosting Few-shot Action Recognition with Graph-guided Hybrid Matching](https://mlsanity.com/paper/2308.09346/) (2023)
- [Use Your Head: Improving Long-Tail Video Recognition](https://mlsanity.com/paper/2304.01143/) (2023)
- [HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding](https://mlsanity.com/paper/2312.03050/) (2023)
- [HR-Pro: Point-supervised Temporal Action Localization via Hierarchical Reliability Propagation](https://mlsanity.com/paper/2308.12608/) (2023)
- [Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition](https://mlsanity.com/paper/2405.19917/) (2024)
- [Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition](https://mlsanity.com/paper/2212.04873/) (2022)
- [CDFSL-V: Cross-Domain Few-Shot Learning for Videos](https://mlsanity.com/paper/2309.03989/) (2023)
- [HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2301.03330/) (2023)
- [One-Shot Action Recognition via Multi-Scale Spatial-Temporal Skeleton Matching](https://mlsanity.com/paper/2307.07286/) (2023)
- [On the Importance of Spatial Relations for Few-shot Action Recognition](https://mlsanity.com/paper/2308.07119/) (2023)
- [A Comprehensive Review of Few-shot Action Recognition](https://mlsanity.com/paper/2407.14744/) (2024)
- [Hierarchical Compositional Representations for Few-shot Action Recognition](https://mlsanity.com/paper/2208.09424/) (2022)
- [On permutation-invariant neural networks](https://mlsanity.com/paper/2403.17410/) (2024)
- [Trajectory-aligned Space-time Tokens for Few-shot Action Recognition](https://mlsanity.com/paper/2407.18249/) (2024)
- [Temporal Alignment-Free Video Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2504.05956/) (2025)
- [Few-shot Action Recognition with Captioning Foundation Models](https://mlsanity.com/paper/2310.10125/) (2023)
- [Augmenting and Aligning Snippets for Few-Shot Video Domain Adaptation](https://mlsanity.com/paper/2303.10451/) (2023)
- [Few-shot Action Recognition via Intra- and Inter-Video Information Maximization](https://mlsanity.com/paper/2305.06114/) (2023)
- [Semantic-aware Video Representation for Few-shot Action Recognition](https://mlsanity.com/paper/2311.06218/) (2023)
- [Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition](https://mlsanity.com/paper/2508.03695/) (2025)
- [MA-FSAR: Multimodal Adaptation of CLIP for Few-Shot Action Recognition](https://mlsanity.com/paper/2308.01532/) (2023)
- [Consistency Prototype Module and Motion Compensation for Few-Shot Action Recognition (CLIP-CPM2C)](https://mlsanity.com/paper/2312.01083/) (2023)
- [Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition](https://mlsanity.com/paper/2402.12706/) (2024)
- [TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognition](https://mlsanity.com/paper/2411.19041/) (2024)
- [Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition](https://mlsanity.com/paper/2602.18043/) (2026)
- [Context-aware Proposal Network for Temporal Action Detection](https://mlsanity.com/paper/2206.09082/) (2022)

## Abstract (arXiv)

Current few-shot action recognition methods reach impressive performance by learning discriminative features for each video via episodic training and designing various temporal alignment strategies. Nevertheless, they are limited in that (a) learning individual features without considering the entire task may lose the most relevant information in the current episode, and (b) these alignment strategies may fail in misaligned instances. To overcome the two limitations, we propose a novel Hybrid Relation guided Set Matching (HyRSM) approach that incorporates two key components: hybrid relation module and set matching metric. The purpose of the hybrid relation module is to learn task-specific embeddings by fully exploiting associated relations within and cross videos in an episode. Built upon the task-specific features, we reformulate distance measure between query and support videos as a set matching problem and further design a bidirectional Mean Hausdorff Metric to improve the resilience to misaligned instances. By this means, the proposed HyRSM can be highly informative and flexible to predict query categories under the few-shot settings. We evaluate HyRSM on six challenging benchmarks, and the experimental results show its superiority over the state-of-the-art methods by a convincing margin. Project page: https://hyrsm-cvpr2022.github.io/.