mlsanityData to 7 Oct 2026

Coherent Temporal Synthesis for Incremental Action Segmentation

arXiv 2403.06102 ↗First posted 10 Mar 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Data replay is a successful incremental learning technique for images. It prevents catastrophic forgetting by keeping a reservoir of previous data, original or synthesized, to ensure the model retains past knowledge while adapting to novel concepts. However, its application in the video domain is rudimentary, as it simply stores frame exemplars for action recognition. This paper presents the first exploration of video data replay techniques for incremental action segmentation, focusing on action temporal modeling. We propose a Temporally Coherent Action (TCA) model, which represents actions using a generative model instead of storing individual frames. The integration of a conditioning variable that captures temporal coherence allows our model to understand the evolution of action features over time. Therefore, action segments generated by TCA for replay are diverse and temporally coherent. In a 10-task incremental setup on the Breakfast dataset, our approach achieves significant increases in accuracy for up to 22% compared to the baselines.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 10 Mar 2024.

Builds on

39 of 55 references.

References · 39

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2016Learning without Forgettingcited by 2,465
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  6. 2017Continual Learning with Deep Generative Replaycited by 1,152
  7. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  8. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  9. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  10. 2019Gradient based sample selection for online continual learningcited by 568
  11. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  12. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
27 more
  1. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  2. 2018Human Action Recognition and Prediction: A Surveycited by 226
  3. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  4. 2021ASFormer: Transformer for Action Segmentationcited by 166
  5. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  6. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  7. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  8. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  9. 2018Generative Models from the perspective of Continual Learningcited by 69
  10. 2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
  11. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  12. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  13. 2021Temporal Action Segmentation from Timestamp Supervisioncited by 48
  14. 2016Weakly supervised learning of actions from transcriptscited by 46
  15. 2022Class-Incremental Learning for Action Recognition in Videoscited by 46
  16. 2020SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentationcited by 39
  17. 2020Set-Constrained Viterbi for Set-Supervised Action Segmentationcited by 35
  18. 2022Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundariescited by 31
  19. 2022vCLIMB: A Novel Video Class Incremental Learning Benchmarkcited by 28
  20. 2022C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentationcited by 23
  21. 2021Iterative Contrast-Classify For Semi-supervised Temporal Action Segmentationcited by 19
  22. 2022Leveraging Action Affinity and Continuity for Semi-supervised Temporal Action Segmentationcited by 17
  23. 2021Temporal Action Segmentation with High-level Complex Activity Labelscited by 16
  24. 2022Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learningcited by 11
  25. 2017Am I Done? Predicting Action Progress in Videoscited by 10
  26. 2023Just a Glimpse: Rethinking Temporal Information for Video Continual Learningcited by 10
  27. 2022Robust Action Segmentation from Timestamp Supervisioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024OnlineTAS: An Online Baseline for Temporal Action Segmentationcited by 7
  2. 2025DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videoscited by 5
  3. 2025Condensing Action Segmentation Datasets via Generative Network Inversioncited by 4
  4. 2025A Temporal Modeling Framework for Video Pre-Training on Video Instance Segmentationcited by 1
  5. 2026Adaptive Latent Trajectory Anchoring for Action Segmentation Dataset Condensationcited by 0