mlsanityData to 14 Oct 2026

Trajectory-aligned Space-time Tokens for Few-shot Action Recognition

arXiv 2407.18249 ↗First posted 25 Jul 2024cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

We propose a simple yet effective approach for few-shot action recognition, emphasizing the disentanglement of motion and appearance representations. By harnessing recent progress in tracking, specifically point trajectories and self-supervised representation learning, we build trajectory-aligned tokens (TATs) that capture motion and appearance information. This approach significantly reduces the data requirements while retaining essential information. To process these representations, we use a Masked Space-time Transformer that effectively learns to aggregate information to facilitate few-shot action recognition. We demonstrate state-of-the-art results on few-shot action recognition across multiple datasets. Our project page is available at https://www.cs.umd.edu/~pulkit/tats

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 54 of its 73 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

54 of 73 references.

References · 54

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2016Matching Networks for One Shot Learningcited by 2,255
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
38 more
  1. 2021ViViT: A Video Vision Transformercited by 1,620
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  4. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  5. 2016Colorful Image Colorizationcited by 1,114
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  7. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  8. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  9. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  10. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  11. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  12. 2023CoTracker: It is Better to Track Togethercited by 522
  13. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 287
  14. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 276
  15. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 263
  16. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 234
  17. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 217
  18. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  19. 2023Tracking Everything Everywhere All at Oncecited by 193
  20. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 181
  21. 2019Few-Shot Video Classification via Temporal Alignmentcited by 125
  22. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  23. 2017Unsupervised Learning by Predicting Noisecited by 98
  24. 2021Temporal-Relational CrossTransformers for Few-Shot Action Recognitioncited by 84
  25. 2023VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimationcited by 82
  26. 2019TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognitioncited by 71
  27. 2017Accurate Optical Flow via Direct Cost Volume Processingcited by 66
  28. 2023MFT: Long-Term Tracking of Every Pixelcited by 62
  29. 2023Dense Optical Tracking: Connecting the Dotscited by 61
  30. 2021Spatio-temporal Relation Modeling for Few-shot Action Recognitioncited by 53
  31. 2020Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognitioncited by 50
  32. 2022Hybrid Relation Guided Set Matching for Few-shot Action Recognitioncited by 50
  33. 2021Self-Supervised Learning for Semi-Supervised Temporal Action Proposalcited by 39
  34. 2023MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognitioncited by 37
  35. 2022Compound Prototype Matching for Few-shot Action Recognitioncited by 35
  36. 2021Learning Implicit Temporal Alignment for Few-shot Video Classificationcited by 27
  37. 2023Boosting Few-shot Action Recognition with Graph-guided Hybrid Matchingcited by 24
  38. 2022Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignmentscited by 23

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 37
  2. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
  3. 2025Temporal Alignment-Free Video Matching for Few-shot Action Recognitioncited by 10
  4. 2025Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognitioncited by 7
  5. 2025Order Matters: On Parameter-Efficient Image-to-Video Probing for Recognizing Nearly Symmetric Actionscited by 3
  6. 2025Beyond Label Semantics: Language-Guided Action Anatomy for Few-shot Action Recognitioncited by 2
  7. 2026Frame2Freq: Spectral Adapters for Fine-Grained Video Understandingcited by 2
  8. 2026TrAction: Action Recognition with Sparse Trajectoriescited by 1
  9. 2026Beyond Instrument Motion: Recognizing Tissue Tension Toward Surgical Skill Assessmentcited by 0
  10. 2026VidParse: Online Parsing of Egocentric Procedures Like a Procited by 0