# Rethinking temporal self-similarity for repetitive action counting

arXiv: https://arxiv.org/abs/2407.09431 · HTML: https://mlsanity.com/paper/2407.09431/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Counting Out Time: Class Agnostic Video Repetition Counting in the Wild](https://mlsanity.com/paper/2006.15418/) (2020)
- [The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video](https://mlsanity.com/paper/2111.09748/) (2021)
- [TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting](https://mlsanity.com/paper/2204.01018/) (2022)
- [Context-aware and Scale-insensitive Temporal Repetition Counting](https://mlsanity.com/paper/2005.08465/) (2020)
- [Full Resolution Repetition Counting](https://mlsanity.com/paper/2305.13778/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Short Note on Evaluating RepNet for Temporal Repetition Counting in Videos](https://mlsanity.com/paper/2411.08878/) (2024)
- [Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activities](https://mlsanity.com/paper/2511.14945/) (2025)
- [TReViS: Temporal Repetition Structure Aware Video Synthesis for Self-supervised Repetitive Action Counting](https://mlsanity.com/paper/2609.24367/) (2026)

## Abstract (arXiv)

Counting repetitive actions in long untrimmed videos is a challenging task that has many applications such as rehabilitation. State-of-the-art methods predict action counts by first generating a temporal self-similarity matrix (TSM) from the sampled frames and then feeding the matrix to a predictor network. The self-similarity matrix, however, is not an optimal input to a network since it discards too much information from the frame-wise embeddings. We thus rethink how a TSM can be utilized for counting repetitive actions and propose a framework that learns embeddings and predicts action start probabilities at full temporal resolution. The number of repeated actions is then inferred from the action start probabilities. In contrast to current approaches that have the TSM as an intermediate representation, we propose a novel loss based on a generated reference TSM, which enforces that the self-similarity of the learned frame-wise embeddings is consistent with the self-similarity of repeated actions. The proposed framework achieves state-of-the-art results on three datasets, i.e., RepCount, UCFRep, and Countix.