# A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark

arXiv: https://arxiv.org/abs/2110.12358 · HTML: https://mlsanity.com/paper/2110.12358/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [TAM: Temporal Adaptive Module for Video Recognition](https://mlsanity.com/paper/2005.06803/) (2020)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Analyzing and Improving Representations with the Soft Nearest Neighbor Loss](https://mlsanity.com/paper/1902.01889/) (2019)
- [Temporal-Relational CrossTransformers for Few-Shot Action Recognition](https://mlsanity.com/paper/2101.06184/) (2021)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Learning to Remember Rare Events](https://mlsanity.com/paper/1703.03129/) (2017)
- [Few-shot Action Recognition with Prototype-centered Attentive Learning](https://mlsanity.com/paper/2101.08085/) (2021)
- [Few-shot Action Recognition with Implicit Temporal Alignment and Pair Similarity Optimization](https://mlsanity.com/paper/2010.06215/) (2020)
- [Few-Shot Action Recognition with Compromised Metric via Optimal Transport](https://mlsanity.com/paper/2104.03737/) (2021)

## Built on by (most cited first, 16 of 16)

- [Compound Prototype Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2207.05515/) (2022)
- [M3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition](https://mlsanity.com/paper/2308.03063/) (2023)
- [Meta-learning approaches for few-shot learning: A survey of recent advances](https://mlsanity.com/paper/2303.07502/) (2023)
- [HyperShot: Few-Shot Learning by Kernel HyperNetworks](https://mlsanity.com/paper/2203.11378/) (2022)
- [Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignments](https://mlsanity.com/paper/2207.10785/) (2022)
- [Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition](https://mlsanity.com/paper/2301.07944/) (2023)
- [CDFSL-V: Cross-Domain Few-Shot Learning for Videos](https://mlsanity.com/paper/2309.03989/) (2023)
- [Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition](https://mlsanity.com/paper/2408.00249/) (2024)
- [Knowledge Prompting for Few-shot Action Recognition](https://mlsanity.com/paper/2211.12030/) (2022)
- [MA-FSAR: Multimodal Adaptation of CLIP for Few-Shot Action Recognition](https://mlsanity.com/paper/2308.01532/) (2023)
- [VLG: General Video Recognition with Web Textual Knowledge](https://mlsanity.com/paper/2212.01638/) (2022)
- [Understanding the Cross-Domain Capabilities of Video-Based Few-Shot Action Recognition Models](https://mlsanity.com/paper/2406.01073/) (2024)
- [Rethinking matching-based few-shot action recognition](https://mlsanity.com/paper/2303.16084/) (2023)
- [LEARN: A Unified Framework for Multi-Task Domain Adapt Few-Shot Learning](https://mlsanity.com/paper/2412.16275/) (2024)
- [Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition](https://mlsanity.com/paper/2505.06002/) (2025)
- [Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs](https://mlsanity.com/paper/2603.26033/) (2026)

## Abstract (arXiv)

The existing few-shot video classification methods often employ a meta-learning paradigm by designing customized temporal alignment module for similarity calculation. While significant progress has been made, these methods fail to focus on learning effective representations, and heavily rely on the ImageNet pre-training, which might be unreasonable for the few-shot recognition setting due to semantics overlap. In this paper, we aim to present an in-depth study on few-shot video classification by making three contributions. First, we perform a consistent comparative study on the existing metric-based methods to figure out their limitations in representation learning. Accordingly, we propose a simple classifier-based baseline without any temporal alignment that surprisingly outperforms the state-of-the-art meta-learning based methods. Second, we discover that there is a high correlation between the novel action class and the ImageNet object class, which is problematic in the few-shot recognition setting. Our results show that the performance of training from scratch drops significantly, which implies that the existing benchmarks cannot provide enough base data. Finally, we present a new benchmark with more base data to facilitate future few-shot video classification without pre-training. The code will be made available at https://github.com/MCG-NJU/FSL-Video.