# On the Importance of Spatial Relations for Few-shot Action Recognition

arXiv: https://arxiv.org/abs/2308.07119 · HTML: https://mlsanity.com/paper/2308.07119/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [CrossTransformers: spatially-aware few-shot transfer](https://mlsanity.com/paper/2007.11498/) (2020)
- [Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference](https://mlsanity.com/paper/2204.07305/) (2022)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Temporal-Relational CrossTransformers for Few-Shot Action Recognition](https://mlsanity.com/paper/2101.06184/) (2021)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Only Time Can Tell: Discovering Temporal Data for Temporal Modeling](https://mlsanity.com/paper/1907.08340/) (2019)
- [HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning](https://mlsanity.com/paper/2201.04182/) (2022)
- [Spatio-temporal Relation Modeling for Few-shot Action Recognition](https://mlsanity.com/paper/2112.05132/) (2021)
- [StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2302.09309/) (2023)

## Built on by (most cited first, 11 of 11)

- [A Comprehensive Review of Few-shot Action Recognition](https://mlsanity.com/paper/2407.14744/) (2024)
- [Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition](https://mlsanity.com/paper/2408.00249/) (2024)
- [Joint Image-Instance Spatial-Temporal Attention for Few-shot Action Recognition](https://mlsanity.com/paper/2503.14430/) (2025)
- [D2ST-Adapter: Disentangled-and-Deformable Spatio-Temporal Adapter for Few-shot Action Recognition](https://mlsanity.com/paper/2312.01431/) (2023)
- [SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition](https://mlsanity.com/paper/2407.16344/) (2024)
- [Video-to-Task Learning via Motion-Guided Attention for Few-Shot Action Recognition](https://mlsanity.com/paper/2411.11335/) (2024)
- [Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition](https://mlsanity.com/paper/2504.10079/) (2025)
- [Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition](https://mlsanity.com/paper/2505.06002/) (2025)
- [A2M2-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition](https://mlsanity.com/paper/2509.17638/) (2025)
- [STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition](https://mlsanity.com/paper/2605.13202/) (2026)
- [Few-Shot Video Recognition via Hierarchical Metric Learning](https://mlsanity.com/paper/2609.05242/) (2026)

## Abstract (arXiv)

Deep learning has achieved great success in video recognition, yet still struggles to recognize novel actions when faced with only a few examples. To tackle this challenge, few-shot action recognition methods have been proposed to transfer knowledge from a source dataset to a novel target dataset with only one or a few labeled videos. However, existing methods mainly focus on modeling the temporal relations between the query and support videos while ignoring the spatial relations. In this paper, we find that the spatial misalignment between objects also occurs in videos, notably more common than the temporal inconsistency. We are thus motivated to investigate the importance of spatial relations and propose a more accurate few-shot action recognition method that leverages both spatial and temporal information. Particularly, a novel Spatial Alignment Cross Transformer (SA-CT) which learns to re-adjust the spatial relations and incorporates the temporal information is contributed. Experiments reveal that, even without using any temporal information, the performance of SA-CT is comparable to temporal based methods on 3/4 benchmarks. To further incorporate the temporal information, we propose a simple yet effective Temporal Mixer module. The Temporal Mixer enhances the video representation and improves the performance of the full SA-CT model, achieving very competitive results. In this work, we also exploit large-scale pretrained models for few-shot action recognition, providing useful insights for this research direction.