# Repetitive Action Counting with Hybrid Temporal Relation Modeling

arXiv: https://arxiv.org/abs/2412.07233 · HTML: https://mlsanity.com/paper/2412.07233/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Benchmarking Micro-action Recognition: Dataset, Methods, and Applications](https://mlsanity.com/paper/2403.05234/) (2024)
- [TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting](https://mlsanity.com/paper/2204.01018/) (2022)
- [LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval](https://mlsanity.com/paper/1909.13784/) (2019)
- [Focal Inverse Distance Transform Maps for Crowd Localization](https://mlsanity.com/paper/2102.07925/) (2021)
- [Context-aware and Scale-insensitive Temporal Repetition Counting](https://mlsanity.com/paper/2005.08465/) (2020)
- [Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition](https://mlsanity.com/paper/2102.07092/) (2021)
- [Language-free Training for Zero-shot Video Grounding](https://mlsanity.com/paper/2210.12977/) (2022)
- [Real-World Repetition Estimation by Div, Grad and Curl](https://mlsanity.com/paper/1802.09971/) (2018)
- [Structure from Recurrent Motion: From Rigidity to Recurrency](https://mlsanity.com/paper/1804.06510/) (2018)
- [TransVCL: Attention-enhanced Video Copy Localization Network with Flexible Supervision](https://mlsanity.com/paper/2211.13090/) (2022)
- [Repetitive Motion Estimation Network: Recover cardiac and respiratory signal from thoracic imaging](https://mlsanity.com/paper/1811.03343/) (2018)

## Built on by (most cited first, 13 of 13)

- [Prototypical Calibrating Ambiguous Samples for Micro-Action Recognition](https://mlsanity.com/paper/2412.14719/) (2024)
- [Online Micro-gesture Recognition Using Data Augmentation and Spatial-Temporal Attention](https://mlsanity.com/paper/2507.09512/) (2025)
- [Motion Matters: Motion-guided Modulation Network for Skeleton-based Micro-Action Recognition](https://mlsanity.com/paper/2507.21977/) (2025)
- [Exploiting Ensemble Learning for Cross-View Isolated Sign Language Recognition](https://mlsanity.com/paper/2502.02196/) (2025)
- [MA-Bench: Towards Fine-grained Micro-Action Understanding](https://mlsanity.com/paper/2603.26586/) (2026)
- [Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion](https://mlsanity.com/paper/2508.15505/) (2025)
- [BVINet: Unlocking Blind Video Inpainting with Zero Annotations](https://mlsanity.com/paper/2502.01181/) (2025)
- [Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation](https://mlsanity.com/paper/2506.12481/) (2025)
- [Prompt-Aware Controllable Shadow Removal](https://mlsanity.com/paper/2501.15043/) (2025)
- [MAC 2026: Advancing Micro-Action Analysis Towards Fine-Grained Understanding](https://mlsanity.com/paper/2607.16284/) (2026)
- [MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition](https://mlsanity.com/paper/2510.10478/) (2025)
- [Unsupervised Discovery of Long-Term Spatiotemporal Periodic Workflows in Human Activities](https://mlsanity.com/paper/2511.14945/) (2025)
- [TReViS: Temporal Repetition Structure Aware Video Synthesis for Self-supervised Repetitive Action Counting](https://mlsanity.com/paper/2609.24367/) (2026)

## Abstract (arXiv)

Repetitive Action Counting (RAC) aims to count the number of repetitive actions occurring in videos. In the real world, repetitive actions have great diversity and bring numerous challenges (e.g., viewpoint changes, non-uniform periods, and action interruptions). Existing methods based on the temporal self-similarity matrix (TSSM) for RAC are trapped in the bottleneck of insufficient capturing action periods when applied to complicated daily videos. To tackle this issue, we propose a novel method named Hybrid Temporal Relation Modeling Network (HTRM-Net) to build diverse TSSM for RAC. The HTRM-Net mainly consists of three key components: bi-modal temporal self-similarity matrix modeling, random matrix dropping, and local temporal context modeling. Specifically, we construct temporal self-similarity matrices by bi-modal (self-attention and dual-softmax) operations, yielding diverse matrix representations from the combination of row-wise and column-wise correlations. To further enhance matrix representations, we propose incorporating a random matrix dropping module to guide channel-wise learning of the matrix explicitly. After that, we inject the local temporal context of video frames and the learned matrix into temporal correlation modeling, which can make the model robust enough to cope with error-prone situations, such as action interruption. Finally, a multi-scale matrix fusion module is designed to aggregate temporal correlations adaptively in multi-scale matrices. Extensive experiments across intra- and cross-datasets demonstrate that the proposed method not only outperforms current state-of-the-art methods but also exhibits robust capabilities in accurately counting repetitive actions in unseen action categories. Notably, our method surpasses the classical TransRAC method by 20.04% in MAE and 22.76% in OBO.