# Live and Learn: Continual Action Clustering with Incremental Views

arXiv: https://arxiv.org/abs/2404.07962 · HTML: https://mlsanity.com/paper/2404.07962/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 29 references that are in the map.
- First posted on arXiv 23 Mar 2024.

## Builds on (most cited first, 3 of 3)

- [Cross-view Action Modeling, Learning and Recognition](https://mlsanity.com/paper/1405.2941/) (2014)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Fast Multi-view Clustering via Ensembles: Towards Scalability, Superiority, and Simplicity](https://mlsanity.com/paper/2203.11572/) (2022)

## Built on by (most cited first, 2 of 2)

- [Association and Consolidation: Evolutionary Memory-Enhanced Incremental Multi-View Clustering](https://mlsanity.com/paper/2509.14544/) (2025)
- [GauSSmart: Enhanced 3D Reconstruction through 2D Foundation Models and Geometric Filtering](https://mlsanity.com/paper/2510.14270/) (2025)

## Abstract (arXiv)

Multi-view action clustering leverages the complementary information from different camera views to enhance the clustering performance. Although existing approaches have achieved significant progress, they assume all camera views are available in advance, which is impractical when the camera view is incremental over time. Besides, learning the invariant information among multiple camera views is still a challenging issue, especially in continual learning scenario. Aiming at these problems, we propose a novel continual action clustering (CAC) method, which is capable of learning action categories in a continual learning manner. To be specific, we first devise a category memory library, which captures and stores the learned categories from historical views. Then, as a new camera view arrives, we only need to maintain a consensus partition matrix, which can be updated by leveraging the incoming new camera view rather than keeping all of them. Finally, a three-step alternate optimization is proposed, in which the category memory library and consensus partition matrix are optimized. The empirical experimental results on 6 realistic multi-view action collections demonstrate the excellent clustering performance and time/space efficiency of the CAC compared with 15 state-of-the-art baselines.