# CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition

arXiv: https://arxiv.org/abs/2004.09215 · HTML: https://mlsanity.com/paper/2004.09215/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2020.

## Built on by (most cited first, 6 of 6)

- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [Class-Incremental Learning for Action Recognition in Videos](https://mlsanity.com/paper/2203.13611/) (2022)
- [Inter-intra Variant Dual Representations forSelf-supervised Video Recognition](https://mlsanity.com/paper/2107.01194/) (2021)
- [TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding](https://mlsanity.com/paper/2110.08814/) (2021)
- [Towards Open-World Gesture Recognition](https://mlsanity.com/paper/2401.11144/) (2024)
- [Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN](https://mlsanity.com/paper/2406.15003/) (2024)

## Abstract (arXiv)

Egocentric gestures are the most natural form of communication for humans to interact with wearable devices such as VR/AR helmets and glasses. A major issue in such scenarios for real-world applications is that may easily become necessary to add new gestures to the system e.g., a proper VR system should allow users to customize gestures incrementally. Traditional deep learning methods require storing all previous class samples in the system and training the model again from scratch by incorporating previous samples and new samples, which costs humongous memory and significantly increases computation over time. In this work, we demonstrate a lifelong 3D convolutional framework -- c(C)la(a)ss increment(t)al net(Net)work (CatNet), which considers temporal information in videos and enables lifelong learning for egocentric gesture video recognition by learning the feature representation of an exemplar set selected from previous class samples. Importantly, we propose a two-stream CatNet, which deploys RGB and depth modalities to train two separate networks. We evaluate CatNets on a publicly available dataset -- EgoGesture dataset, and show that CatNets can learn many classes incrementally over a long period of time. Results also demonstrate that the two-stream architecture achieves the best performance on both joint training and class incremental training compared to 3 other one-stream architectures. The codes and pre-trained models used in this work are provided at https://github.com/villawang/CatNet.