# Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

arXiv: https://arxiv.org/abs/2203.14957 · HTML: https://mlsanity.com/paper/2203.14957/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 49 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [SpeedNet: Learning the Speediness in Videos](https://mlsanity.com/paper/2004.06130/) (2020)
- [Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation](https://mlsanity.com/paper/1707.03374/) (2017)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)

## Built on by (most cited first, 27 of 27)

- [Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment](https://mlsanity.com/paper/2306.05526/) (2023)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space](https://mlsanity.com/paper/2206.11895/) (2022)
- [SEAL: Semantic Attention Learning for Long Video Representation](https://mlsanity.com/paper/2412.01798/) (2024)
- [Heterogeneous Skeleton-Based Action Representation Learning](https://mlsanity.com/paper/2506.03481/) (2025)
- [Weakly Supervised Video Representation Learning with Unaligned Text for Sequential Videos](https://mlsanity.com/paper/2303.12370/) (2023)
- [F3Set: Towards Analyzing Fast, Frequent, and Fine-grained Events from Videos](https://mlsanity.com/paper/2504.08222/) (2025)
- [Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations](https://mlsanity.com/paper/2503.19706/) (2025)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [CoachMe: Decoding Sport Elements with a Reference-Based Coaching Instruction Generation Model](https://mlsanity.com/paper/2509.11698/) (2025)
- [Weakly-supervised Representation Learning for Video Alignment and Analysis](https://mlsanity.com/paper/2302.04064/) (2023)
- [Leveraging Superfluous Information in Contrastive Representation Learning](https://mlsanity.com/paper/2408.10292/) (2024)
- [FinePseudo: Improving Pseudo-Labelling through Temporal-Alignablity for Semi-Supervised Fine-Grained Action Recognition](https://mlsanity.com/paper/2409.01448/) (2024)
- [Synchronization of Multiple Videos](https://mlsanity.com/paper/2510.14051/) (2025)
- [Sequential Contrastive Audio-Visual Learning](https://mlsanity.com/paper/2407.05782/) (2024)
- [Self-supervised and Weakly Supervised Contrastive Learning for Frame-wise Action Representations](https://mlsanity.com/paper/2212.03125/) (2022)
- [How Effective are Self-Supervised Models for Contact Identification in Videos](https://mlsanity.com/paper/2408.00498/) (2024)
- [Sync from the Sea: Retrieving Alignable Videos from Large-Scale Datasets](https://mlsanity.com/paper/2409.01445/) (2024)
- [Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment](https://mlsanity.com/paper/2409.04607/) (2024)
- [Beyond Audio and Pose: A General-Purpose Framework for Video Synchronization](https://mlsanity.com/paper/2506.15937/) (2025)
- [End to End AI System for Surgical Gesture Sequence Recognition and Clinical Outcome Prediction](https://mlsanity.com/paper/2511.11899/) (2025)
- [Multi-entity Video Transformers for Fine-Grained Video Representation Learning](https://mlsanity.com/paper/2311.10873/) (2023)
- [A Causality-Inspired Model for Intima-Media Thickening Assessment in Ultrasound Videos](https://mlsanity.com/paper/2503.12418/) (2025)
- [Multi Activity Sequence Alignment via Implicit Clustering](https://mlsanity.com/paper/2503.12519/) (2025)
- [Spatial-Temporal Pre-Training for Embryo Viability Prediction Using Time-Lapse Videos](https://mlsanity.com/paper/2506.17403/) (2025)
- [PRISM: Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning](https://mlsanity.com/paper/2608.30388/) (2026)

## Abstract (arXiv)

Prior works on action representation learning mainly focus on designing various architectures to extract the global representations for short video clips. In contrast, many practical applications such as video alignment have strong demand for learning dense representations for long videos. In this paper, we introduce a novel contrastive action representation learning (CARL) framework to learn frame-wise action representations, especially for long videos, in a self-supervised manner. Concretely, we introduce a simple yet efficient video encoder that considers spatio-temporal context to extract frame-wise representations. Inspired by the recent progress of self-supervised learning, we present a novel sequence contrastive loss (SCL) applied on two correlated views obtained through a series of spatio-temporal data augmentations. SCL optimizes the embedding space by minimizing the KL-divergence between the sequence similarity of two augmented views and a prior Gaussian distribution of timestamp distance. Experiments on FineGym, PennAction and Pouring datasets show that our method outperforms previous state-of-the-art by a large margin for downstream fine-grained action classification. Surprisingly, although without training on paired videos, our approach also shows outstanding performance on video alignment and fine-grained frame retrieval tasks. Code and models are available at https://github.com/minghchen/CARL_code.