mlsanityData to 14 Oct 2026

POET: Prompt Offset Tuning for Continual Human Action Adaptation

arXiv 2504.18059 ↗First posted 25 Apr 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

As extended reality (XR) is redefining how users interact with computing devices, research in human action recognition is gaining prominence. Typically, models deployed on immersive computing devices are static and limited to their default set of classes. The goal of our research is to provide users and developers with the capability to personalize their experience by adding new action classes to their device models continually. Importantly, a user should be able to add new classes in a low-shot and efficient manner, while this process should not require storing or replaying any of user's sensitive training data. We formalize this problem as privacy-aware few-shot continual action recognition. Towards this end, we propose POET: Prompt-Offset Tuning. While existing prompt tuning approaches have shown great promise for continual learning of image, text, and video modalities; they demand access to extensively pretrained transformers. Breaking away from this assumption, POET demonstrates the efficacy of prompt tuning a significantly lightweight backbone, pretrained exclusively on the base class data. We propose a novel spatio-temporal learnable prompt offset tuning approach, and are the first to apply such prompt tuning to Graph Neural Networks. We contribute two new benchmarks for our new problem setting in human action recognition: (i) NTU RGB+D dataset for activity recognition, and (ii) SHREC-2017 dataset for hand gesture recognition. We find that POET consistently outperforms comprehensive benchmarks. Source code at https://github.com/humansensinglab/POET-continual-action-recognition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 58 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

48 of 58 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  7. 2016Learning without Forgettingcited by 2,496
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2016Progressive Neural Networkscited by 1,361
  11. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  12. 2021Learning to Prompt for Continual Learningcited by 944
36 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  2. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  3. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  4. 2020A Generalization of Transformer Networks to Graphscited by 661
  5. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  6. 2021ImageNet-21K Pretraining for the Massescited by 613
  7. 2020Jukebox: A Generative Model for Musiccited by 542
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  9. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  10. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  11. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 333
  12. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 322
  13. 2020Few-Shot Class-Incremental Learningcited by 251
  14. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  15. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  16. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  17. 2021GraphiT: Encoding Graph Structure in Transformerscited by 147
  18. 2022Forward Compatible Few-Shot Class-Incremental Learningcited by 127
  19. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  20. 2022Constrained Few-shot Class-incremental Learningcited by 94
  21. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  22. 2022Few-Shot Class-Incremental Learning from an Open-Set Perspectivecited by 84
  23. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  24. 2020Hierarchical Quantized Autoencoderscited by 57
  25. 2023Online Clustered Codebookcited by 55
  26. 2020A Survey on 3D Skeleton-Based Action Recognition Using Learning Methodcited by 47
  27. 2022PIVOT: Prompting for Video Continual Learningcited by 45
  28. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  29. 2023MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognitioncited by 37
  30. 2020Class-incremental Learning with Pre-allocated Fixed Classifierscited by 33
  31. 2023STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognitioncited by 29
  32. 2019Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attentioncited by 22
  33. 2023`A-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Promptingcited by 17
  34. 2022PrivHAR: Recognizing Human Actions From Privacy-preserving Lenscited by 16
  35. 2022Privacy-Preserving Action Recognition via Motion Difference Quantizationcited by 14
  36. 2022Adaptive Local-Component-aware Graph Convolutional Network for One-shot Skeleton-based Action Recognitioncited by 9

Built on by

Not cited yet.