# Temporally Coherent Embeddings for Self-Supervised Video Representation Learning

arXiv: https://arxiv.org/abs/2004.02753 · HTML: https://mlsanity.com/paper/2004.02753/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2020.

## Built on by (most cited first, 17 of 17)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds](https://mlsanity.com/paper/2109.00179/) (2021)
- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast](https://mlsanity.com/paper/2106.03796/) (2021)
- [Self-supervised video pretraining yields robust and more human-aligned visual representations](https://mlsanity.com/paper/2210.06433/) (2022)
- [Time to augment self-supervised visual representation learning](https://mlsanity.com/paper/2207.13492/) (2022)
- [Self-supervised visual learning from interactions with objects](https://mlsanity.com/paper/2407.06704/) (2024)
- [Self-supervised Learning for Unintentional Action Prediction](https://mlsanity.com/paper/2209.12074/) (2022)
- [Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learning](https://mlsanity.com/paper/2209.10901/) (2022)
- [Learning Object Semantic Similarity with Self-Supervision](https://mlsanity.com/paper/2405.05143/) (2024)
- [SVGraph: Learning Semantic Graphs from Instructional Videos](https://mlsanity.com/paper/2207.08001/) (2022)
- [Case-Based Inverse Reinforcement Learning Using Temporal Coherence](https://mlsanity.com/paper/2206.05827/) (2022)
- [How Effective are Self-Supervised Models for Contact Identification in Videos](https://mlsanity.com/paper/2408.00498/) (2024)
- [Embodied vision for learning object representations](https://mlsanity.com/paper/2205.06198/) (2022)
- [Multi-encoder attention-based architectures for sound recognition with partial visual assistance](https://mlsanity.com/paper/2209.12826/) (2022)
- [KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder](https://mlsanity.com/paper/2411.12270/) (2024)

## Abstract (arXiv)

This paper presents TCE: Temporally Coherent Embeddings for self-supervised video representation learning. The proposed method exploits inherent structure of unlabeled video data to explicitly enforce temporal coherency in the embedding space, rather than indirectly learning it through ranking or predictive proxy tasks. In the same way that high-level visual information in the world changes smoothly, we believe that nearby frames in learned representations will benefit from demonstrating similar properties. Using this assumption, we train our TCE model to encode videos such that adjacent frames exist close to each other and videos are separated from one another. Using TCE we learn robust representations from large quantities of unlabeled video data. We thoroughly analyse and evaluate our self-supervised learned TCE models on a downstream task of video action recognition using multiple challenging benchmarks (Kinetics400, UCF101, HMDB51). With a simple but effective 2D-CNN backbone and only RGB stream inputs, TCE pre-trained representations outperform all previous selfsupervised 2D-CNN and 3D-CNN pre-trained on UCF101. The code and pre-trained models for this paper can be downloaded at: https://github.com/csiro-robotics/TCE