# A2M2-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition

arXiv: https://arxiv.org/abs/2509.17638 · HTML: https://mlsanity.com/paper/2509.17638/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 55 of its 89 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [CrossTransformers: spatially-aware few-shot transfer](https://mlsanity.com/paper/2007.11498/) (2020)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Temporal-Relational CrossTransformers for Few-Shot Action Recognition](https://mlsanity.com/paper/2101.06184/) (2021)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization](https://mlsanity.com/paper/1712.01034/) (2017)
- [Dense Classification and Implanting for Few-Shot Learning](https://mlsanity.com/paper/1903.05050/) (2019)
- [Spatio-temporal Relation Modeling for Few-shot Action Recognition](https://mlsanity.com/paper/2112.05132/) (2021)
- [Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification](https://mlsanity.com/paper/2204.04567/) (2022)
- [Hybrid Relation Guided Set Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2204.13423/) (2022)
- [Is Second-order Information Helpful for Large-scale Visual Recognition?](https://mlsanity.com/paper/1703.08050/) (2017)

## Abstract (arXiv)

Thanks to capability to alleviate the cost of large-scale annotation, few-shot action recognition (FSAR) has attracted increased attention of researchers in recent years. Existing FSAR approaches typically neglect the role of individual motion pattern in comparison, and under-explore the feature statistics for video dynamics. Thereby, they struggle to handle the challenging temporal misalignment in video dynamics, particularly by using 2D backbones. To overcome these limitations, this work proposes an adaptively aligned multi-scale second-order moment network, namely A2M2-Net, to describe the latent video dynamics with a collection of powerful representation candidates and adaptively align them in an instance-guided manner. To this end, our A2M2-Net involves two core components, namely, adaptive alignment (A2 module) for matching, and multi-scale second-order moment (M2 block) for strong representation. Specifically, M2 block develops a collection of semantic second-order descriptors at multiple spatio-temporal scales. Furthermore, A2 module aims to adaptively select informative candidate descriptors while considering the individual motion pattern. By such means, our A2M2-Net is able to handle the challenging temporal misalignment problem by establishing an adaptive alignment protocol for strong representation. Notably, our proposed method generalizes well to various few-shot settings and diverse metrics. The experiments are conducted on five widely used FSAR benchmarks, and the results show our A2M2-Net achieves very competitive performance compared to state-of-the-arts, demonstrating its effectiveness and generalization.