# TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning

arXiv: https://arxiv.org/abs/2112.03587 · HTML: https://mlsanity.com/paper/2112.03587/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 115 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 30 of 69)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [FcaNet: Frequency Channel Attention Networks](https://mlsanity.com/paper/2012.11879/) (2020)
- [Colorization as a Proxy Task for Visual Understanding](https://mlsanity.com/paper/1703.04044/) (2017)

## Built on by (most cited first, 30 of 31)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context](https://mlsanity.com/paper/2207.08132/) (2022)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning](https://mlsanity.com/paper/2212.05330/) (2022)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Visual Causal Scene Refinement for Video Question Answering](https://mlsanity.com/paper/2305.04224/) (2023)
- [Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2211.10412/) (2022)
- [Urban Regional Function Guided Traffic Flow Prediction](https://mlsanity.com/paper/2303.09789/) (2023)
- [Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos](https://mlsanity.com/paper/2209.05653/) (2022)
- [Adaptive Graph Convolutional Networks for Weakly Supervised Anomaly Detection in Videos](https://mlsanity.com/paper/2202.06503/) (2022)
- [DenseLight: Efficient Control for Large-scale Traffic Signals with Dense Feedback](https://mlsanity.com/paper/2306.07553/) (2023)
- [CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning](https://mlsanity.com/paper/2306.17462/) (2023)
- [MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments](https://mlsanity.com/paper/2402.00290/) (2024)
- [VideoPure: Diffusion-based Adversarial Purification for Video Recognition](https://mlsanity.com/paper/2501.14999/) (2025)
- [Self-supervised Contrastive Learning for Audio-Visual Action Recognition](https://mlsanity.com/paper/2204.13386/) (2022)
- [VCD: Visual Causality Discovery for Cross-Modal Question Reasoning](https://mlsanity.com/paper/2304.08083/) (2023)
- [Cross-Stream Contrastive Learning for Self-Supervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2305.02324/) (2023)
- [Building Category Graphs Representation with Spatial and Temporal Attention for Visual Navigation](https://mlsanity.com/paper/2312.03327/) (2023)
- [Learning to Visually Connect Actions and their Effects](https://mlsanity.com/paper/2401.10805/) (2024)
- [GOCA: Guided Online Cluster Assignment for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2207.10158/) (2022)
- [A baseline on continual learning methods for video action recognition](https://mlsanity.com/paper/2304.10335/) (2023)
- [Lightweight Model Pre-training via Language Guided Knowledge Distillation](https://mlsanity.com/paper/2406.11689/) (2024)
- [Collaboratively Self-supervised Video Representation Learning for Action Recognition](https://mlsanity.com/paper/2401.07584/) (2024)
- [DRDM: A Disentangled Representations Diffusion Model for Synthesizing Realistic Person Images](https://mlsanity.com/paper/2412.18797/) (2024)
- [Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup](https://mlsanity.com/paper/2503.02284/) (2025)
- [Towards an Effective Action-Region Tracking Framework for Fine-grained Video Action Recognition](https://mlsanity.com/paper/2511.21202/) (2025)
- [Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localization](https://mlsanity.com/paper/2512.16504/) (2025)
- [A Scalable Curiosity-Driven Game-Theoretic Framework for Long-Tail Multi-Label Learning in Data Mining](https://mlsanity.com/paper/2602.15330/) (2026)

## Abstract (arXiv)

Video self-supervised learning is a challenging task, which requires significant expressive power from the model to leverage rich spatial-temporal knowledge and generate effective supervisory signals from large amounts of unlabeled videos. However, existing methods fail to increase the temporal diversity of unlabeled videos and ignore elaborately modeling multi-scale temporal dependencies in an explicit way. To overcome these limitations, we take advantage of the multi-scale temporal dependencies within videos and proposes a novel video self-supervised learning framework named Temporal Contrastive Graph Learning (TCGL), which jointly models the inter-snippet and intra-snippet temporal dependencies for temporal representation learning with a hybrid graph contrastive learning strategy. Specifically, a Spatial-Temporal Knowledge Discovering (STKD) module is first introduced to extract motion-enhanced spatial-temporal representations from videos based on the frequency domain analysis of discrete cosine transform. To explicitly model multi-scale temporal dependencies of unlabeled videos, our TCGL integrates the prior knowledge about the frame and snippet orders into graph structures, i.e., the intra-/inter- snippet Temporal Contrastive Graphs (TCG). Then, specific contrastive learning modules are designed to maximize the agreement between nodes in different graph views. To generate supervisory signals for unlabeled videos, we introduce an Adaptive Snippet Order Prediction (ASOP) module which leverages the relational knowledge among video snippets to learn the global context representation and recalibrate the channel-wise features adaptively. Experimental results demonstrate the superiority of our TCGL over the state-of-the-art methods on large-scale action recognition and video retrieval benchmarks.The code is publicly available at https://github.com/YangLiu9208/TCGL.