mlsanityData to 7 Oct 2026

Task-Specific Alignment and Multiple Level Transformer for Few-Shot Action Recognition

arXiv 2307.01985 ↗First posted 5 Jul 2023cs.CV cs.DMcited by 2

Not on the map. Why?

Abstract (arXiv)

In the research field of few-shot learning, the main difference between image-based and video-based is the additional temporal dimension. In recent years, some works have used the Transformer to deal with frames, then get the attention feature and the enhanced prototype, and the results are competitive. However, some video frames may relate little to the action, and only using single frame-level or segment-level features may not mine enough information. We address these problems sequentially through an end-to-end method named "Task-Specific Alignment and Multiple-level Transformer Network (TSA-MLT)". The first module (TSA) aims at filtering the action-irrelevant frames for action duration alignment. Affine Transformation for frame sequence in the time dimension is used for linear sampling. The second module (MLT) focuses on the Multiple-level feature of the support prototype and query sample to mine more information for the alignment, which operates on different level features. We adopt a fusion loss according to a fusion distance that fuses the L2 sequence distance, which focuses on temporal order alignment, and the Optimal Transport distance, which focuses on measuring the gap between the appearance and semantics of the videos. Extensive experiments show our method achieves state-of-the-art results on the HMDB51 and UCF101 datasets and a competitive result on the benchmark of Kinetics and something 2-something V2 datasets. Our code is available at the URL: https://github.com/cofly2014/tsa-mlt.git

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 65 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

32 of 65 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Layer Normalizationcited by 5,579
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  7. 2016Matching Networks for One Shot Learningcited by 2,246
  8. 2015Spatial Transformer Networkscited by 1,423
  9. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
20 more
  1. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  2. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  3. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  4. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 208
  5. 2019Few-Shot Video Classification via Temporal Alignmentcited by 124
  6. 2017Learning to Compose Domain-Specific Transformations for Data Augmentationcited by 91
  7. 2021Temporal-Relational CrossTransformers for Few-Shot Action Recognitioncited by 83
  8. 2019TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognitioncited by 71
  9. 2021Spatio-temporal Relation Modeling for Few-shot Action Recognitioncited by 52
  10. 2020Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognitioncited by 50
  11. 2022Hybrid Relation Guided Set Matching for Few-shot Action Recognitioncited by 49
  12. 2020Multimodal Prototypical Networks for Few-shot Learningcited by 48
  13. 2019ProtoGAN: Towards Few Shot Learning for Action Recognitioncited by 41
  14. 2023MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognitioncited by 36
  15. 2021TA2N: Two-Stage Action Alignment Network for Few-shot Action Recognitioncited by 35
  16. 2021Few-shot Action Recognition with Prototype-centered Attentive Learningcited by 28
  17. 2022Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignmentscited by 22
  18. 2020TAEN: Temporal Aware Embedding Network for Few-Shot Action Recognitioncited by 12
  19. 2021Few-Shot Action Recognition with Compromised Metric via Optimal Transportcited by 12
  20. 2022Hierarchical Compositional Representations for Few-shot Action Recognitioncited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024DMSD-CDFSAR: Distillation from Mixed-Source Domain for Cross-Domain Few-shot Action Recognitioncited by 3
  2. 2025Joint Image-Instance Spatial-Temporal Attention for Few-shot Action Recognitioncited by 2