mlsanityData to 7 Oct 2026

On the Importance of Spatial Relations for Few-shot Action Recognition

arXiv 2308.07119 ↗First posted 14 Aug 2023cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Deep learning has achieved great success in video recognition, yet still struggles to recognize novel actions when faced with only a few examples. To tackle this challenge, few-shot action recognition methods have been proposed to transfer knowledge from a source dataset to a novel target dataset with only one or a few labeled videos. However, existing methods mainly focus on modeling the temporal relations between the query and support videos while ignoring the spatial relations. In this paper, we find that the spatial misalignment between objects also occurs in videos, notably more common than the temporal inconsistency. We are thus motivated to investigate the importance of spatial relations and propose a more accurate few-shot action recognition method that leverages both spatial and temporal information. Particularly, a novel Spatial Alignment Cross Transformer (SA-CT) which learns to re-adjust the spatial relations and incorporates the temporal information is contributed. Experiments reveal that, even without using any temporal information, the performance of SA-CT is comparable to temporal based methods on 3/4 benchmarks. To further incorporate the temporal information, we propose a simple yet effective Temporal Mixer module. The Temporal Mixer enhances the video representation and improves the performance of the full SA-CT model, achieving very competitive results. In this work, we also exploit large-scale pretrained models for few-shot action recognition, providing useful insights for this research direction.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016Layer Normalizationcited by 5,579
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  12. 2016Matching Networks for One Shot Learningcited by 2,246
27 more
  1. 2018SlowFast Networks for Video Recognitioncited by 1,862
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  6. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  7. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  8. 2017Meta Networkscited by 268
  9. 2019Cross Attention Network for Few-shot Classificationcited by 218
  10. 2020CrossTransformers: spatially-aware few-shot transfercited by 174
  11. 2019Few-Shot Video Classification via Temporal Alignmentcited by 124
  12. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 123
  13. 2021Temporal-Relational CrossTransformers for Few-Shot Action Recognitioncited by 83
  14. 2019TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognitioncited by 71
  15. 2019Only Time Can Tell: Discovering Temporal Data for Temporal Modelingcited by 66
  16. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  17. 2023StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learningcited by 53
  18. 2021Spatio-temporal Relation Modeling for Few-shot Action Recognitioncited by 52
  19. 2020Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognitioncited by 50
  20. 2022Hybrid Relation Guided Set Matching for Few-shot Action Recognitioncited by 49
  21. 2021Shallow Bayesian Meta Learning for Real-World Few-Shot Recognitioncited by 48
  22. 2021Meta-FDMixup: Cross-Domain Few-Shot Learning Guided by Labeled Target Datacited by 48
  23. 2019ProtoGAN: Towards Few Shot Learning for Action Recognitioncited by 41
  24. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
  25. 2022TGDM: Target Guided Dynamic Mixup for Cross-Domain Few-Shot Learningcited by 16
  26. 2022Semantic-aligned Fusion Transformer for One-shot Object Detectioncited by 13
  27. 2023DETA: Denoised Task Adaptation for Few-Shot Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 2.

11 papers.

  1. 2024A Comprehensive Review of Few-shot Action Recognitioncited by 11
  2. 2024Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognitioncited by 8
  3. 2025Joint Image-Instance Spatial-Temporal Attention for Few-shot Action Recognitioncited by 2
  4. 2023D2ST-Adapter: Disentangled-and-Deformable Spatio-Temporal Adapter for Few-shot Action Recognitioncited by 1
  5. 2024SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognitioncited by 1
  6. 2024Video-to-Task Learning via Motion-Guided Attention for Few-Shot Action Recognitioncited by 0
  7. 2025Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognitioncited by 0
  8. 2025Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognitioncited by 0
  9. 2025A2M2-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognitioncited by 0
  10. 2026STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognitioncited by 0
  11. 2026Few-Shot Video Recognition via Hierarchical Metric Learningcited by 0