# Video Representation Learning with Visual Tempo Consistency

arXiv: https://arxiv.org/abs/2006.15489 · HTML: https://mlsanity.com/paper/2006.15489/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 30 of 57)

- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [Instance Localization for Self-supervised Detection Pretraining](https://mlsanity.com/paper/2102.08318/) (2021)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [SVFormer: Semi-supervised Video Transformer for Action Recognition](https://mlsanity.com/paper/2211.13222/) (2022)
- [Semi-Supervised Action Recognition with Temporal Contrastive Learning](https://mlsanity.com/paper/2102.02751/) (2021)
- [Time-Equivariant Contrastive Video Representation Learning](https://mlsanity.com/paper/2112.03624/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Self-supervised Video Representation Learning by Context and Motion Decoupling](https://mlsanity.com/paper/2104.00862/) (2021)
- [Multiview Pseudo-Labeling for Semi-supervised Learning from Video](https://mlsanity.com/paper/2104.00682/) (2021)
- [Unsupervised Pre-training for Temporal Action Localization Tasks](https://mlsanity.com/paper/2203.13609/) (2022)
- [Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition](https://mlsanity.com/paper/2112.09690/) (2021)
- [VideoMamba: Spatio-Temporal Selective State Space Model](https://mlsanity.com/paper/2407.08476/) (2024)
- [Motion-Focused Contrastive Learning of Video Representations](https://mlsanity.com/paper/2201.04029/) (2022)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [Static and Dynamic Concepts for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2207.12795/) (2022)
- [Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity](https://mlsanity.com/paper/2112.05883/) (2021)
- [Unsupervised Visual Representation Learning by Tracking Patches in Video](https://mlsanity.com/paper/2105.02545/) (2021)
- [Self-supervised video pretraining yields robust and more human-aligned visual representations](https://mlsanity.com/paper/2210.06433/) (2022)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [Capturing Co-existing Distortions in User-Generated Content for No-reference Video Quality Assessment](https://mlsanity.com/paper/2307.16813/) (2023)
- [Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision](https://mlsanity.com/paper/2112.05181/) (2021)
- [Composable Augmentation Encoding for Video Representation Learning](https://mlsanity.com/paper/2104.00616/) (2021)
- [Transcript to Video: Efficient Clip Sequencing from Texts](https://mlsanity.com/paper/2107.11851/) (2021)
- [Cross-Architecture Self-supervised Video Representation Learning](https://mlsanity.com/paper/2205.13313/) (2022)

## Abstract (arXiv)

Visual tempo, which describes how fast an action goes, has shown its potential in supervised action recognition. In this work, we demonstrate that visual tempo can also serve as a self-supervision signal for video representation learning. We propose to maximize the mutual information between representations of slow and fast videos via hierarchical contrastive learning (VTHCL). Specifically, by sampling the same instance at slow and fast frame rates respectively, we can obtain slow and fast video frames which share the same semantics but contain different visual tempos. Video representations learned from VTHCL achieve the competitive performances under the self-supervision evaluation protocol for action recognition on UCF-101 (82.1%) and HMDB-51 (49.2%). Moreover, comprehensive experiments suggest that the learned representations are generalized well to other downstream tasks including action detection on AVA and action anticipation on Epic-Kitchen. Finally, we propose Instance Correspondence Map (ICM) to visualize the shared semantics captured by contrastive learning.