# Few-Shot Action Recognition with Compromised Metric via Optimal Transport

arXiv: https://arxiv.org/abs/2104.03737 · HTML: https://mlsanity.com/paper/2104.03737/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation](https://mlsanity.com/paper/1910.13616/) (2019)
- [Finding Task-Relevant Features for Few-Shot Learning by Category Traversal](https://mlsanity.com/paper/1905.11116/) (2019)
- [CRNet: Cross-Reference Networks for Few-Shot Segmentation](https://mlsanity.com/paper/2003.10658/) (2020)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Going Deeper into Action Recognition: A Survey](https://mlsanity.com/paper/1605.04988/) (2016)
- [Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition](https://mlsanity.com/paper/2010.09982/) (2020)
- [Metalearned Neural Memory](https://mlsanity.com/paper/1907.09720/) (2019)
- [ProtoGAN: Towards Few Shot Learning for Action Recognition](https://mlsanity.com/paper/1909.07945/) (2019)
- [Few-shot Action Recognition with Prototype-centered Attentive Learning](https://mlsanity.com/paper/2101.08085/) (2021)
- [Few-shot Action Recognition with Implicit Temporal Alignment and Pair Similarity Optimization](https://mlsanity.com/paper/2010.06215/) (2020)
- [TAEN: Temporal Aware Embedding Network for Few-Shot Action Recognition](https://mlsanity.com/paper/2004.10141/) (2020)

## Built on by (most cited first, 12 of 12)

- [Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignments](https://mlsanity.com/paper/2207.10785/) (2022)
- [A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark](https://mlsanity.com/paper/2110.12358/) (2021)
- [Hierarchical Compositional Representations for Few-shot Action Recognition](https://mlsanity.com/paper/2208.09424/) (2022)
- [Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition](https://mlsanity.com/paper/2408.12475/) (2024)
- [Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition](https://mlsanity.com/paper/2408.00249/) (2024)
- [Consistency Prototype Module and Motion Compensation for Few-Shot Action Recognition (CLIP-CPM2C)](https://mlsanity.com/paper/2312.01083/) (2023)
- [DMSD-CDFSAR: Distillation from Mixed-Source Domain for Cross-Domain Few-shot Action Recognition](https://mlsanity.com/paper/2407.05657/) (2024)
- [Anomaly Crossing: New Horizons for Video Anomaly Detection as Cross-domain Few-shot Learning](https://mlsanity.com/paper/2112.06320/) (2021)
- [Task-Specific Alignment and Multiple Level Transformer for Few-Shot Action Recognition](https://mlsanity.com/paper/2307.01985/) (2023)
- [Two-stream joint matching method based on contrastive learning for few-shot action recognition](https://mlsanity.com/paper/2401.04150/) (2024)
- [Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition](https://mlsanity.com/paper/2505.06002/) (2025)
- [A2M2-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition](https://mlsanity.com/paper/2509.17638/) (2025)

## Abstract (arXiv)

Although vital to computer vision systems, few-shot action recognition is still not mature despite the wide research of few-shot image classification. Popular few-shot learning algorithms extract a transferable embedding from seen classes and reuse it on unseen classes by constructing a metric-based classifier. One main obstacle to applying these algorithms in action recognition is the complex structure of videos. Some existing solutions sample frames from a video and aggregate their embeddings to form a video-level representation, neglecting important temporal relations. Others perform an explicit sequence matching between two videos and define their distance as matching cost, imposing too strong restrictions on sequence ordering. In this paper, we propose Compromised Metric via Optimal Transport (CMOT) to combine the advantages of these two solutions. CMOT simultaneously considers semantic and temporal information in videos under Optimal Transport framework, and is discriminative for both content-sensitive and ordering-sensitive tasks. In detail, given two videos, we sample segments from them and cast the calculation of their distance as an optimal transport problem between two segment sequences. To preserve the inherent temporal ordering information, we additionally amend the ground cost matrix by penalizing it with the positional distance between a pair of segments. Empirical results on benchmark datasets demonstrate the superiority of CMOT.