# SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning

arXiv: https://arxiv.org/abs/2008.00975 · HTML: https://mlsanity.com/paper/2008.00975/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2020.

## Built on by (most cited first, 30 of 65)

- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective](https://mlsanity.com/paper/2103.17263/) (2021)
- [SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving](https://mlsanity.com/paper/2106.11118/) (2021)
- [Long-Short Temporal Contrastive Learning of Video Transformers](https://mlsanity.com/paper/2106.09212/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Probabilistic Representations for Video Contrastive Learning](https://mlsanity.com/paper/2204.03946/) (2022)
- [A Style and Semantic Memory Mechanism for Domain Generalization](https://mlsanity.com/paper/2112.07517/) (2021)
- [TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2112.03587/) (2021)
- [MAR: Masked Autoencoders for Efficient Action Recognition](https://mlsanity.com/paper/2207.11660/) (2022)
- [Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization](https://mlsanity.com/paper/2108.02183/) (2021)
- [Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning](https://mlsanity.com/paper/2203.14957/) (2022)
- [Self-supervised Representation Learning Framework for Remote Physiological Measurement Using Spatiotemporal Augmentation Loss](https://mlsanity.com/paper/2107.07695/) (2021)
- [Motion-Focused Contrastive Learning of Video Representations](https://mlsanity.com/paper/2201.04029/) (2022)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [Static and Dynamic Concepts for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2207.12795/) (2022)
- [DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks](https://mlsanity.com/paper/2304.00571/) (2023)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [Stand-Alone Inter-Frame Attention in Video Models](https://mlsanity.com/paper/2206.06931/) (2022)
- [CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising](https://mlsanity.com/paper/2112.07515/) (2021)
- [SLIC: Self-Supervised Learning with Iterative Clustering for Human Action Videos](https://mlsanity.com/paper/2206.12534/) (2022)
- [TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition](https://mlsanity.com/paper/2303.16268/) (2023)
- [Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision](https://mlsanity.com/paper/2112.05181/) (2021)
- [Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency](https://mlsanity.com/paper/2204.03017/) (2022)
- [Dual Contrastive Learning for Spatio-temporal Representation](https://mlsanity.com/paper/2207.05340/) (2022)
- [Transferrable Contrastive Learning for Visual Domain Adaptation](https://mlsanity.com/paper/2112.07516/) (2021)
- [SIGMA: Sinkhorn-Guided Masked Video Modeling](https://mlsanity.com/paper/2407.15447/) (2024)
- [Contrastive Spatio-Temporal Pretext Learning for Self-supervised Video Representation](https://mlsanity.com/paper/2112.08913/) (2021)
- [Motion Sensitive Contrastive Learning for Self-supervised Video Representation](https://mlsanity.com/paper/2208.06105/) (2022)

## Abstract (arXiv)

A steady momentum of innovations and breakthroughs has convincingly pushed the limits of unsupervised image representation learning. Compared to static 2D images, video has one more dimension (time). The inherent supervision existing in such sequential structure offers a fertile ground for building unsupervised learning models. In this paper, we compose a trilogy of exploring the basic and generic supervision in the sequence from spatial, spatiotemporal and sequential perspectives. We materialize the supervisory signals through determining whether a pair of samples is from one frame or from one video, and whether a triplet of samples is in the correct temporal order. We uniquely regard the signals as the foundation in contrastive learning and derive a particular form named Sequence Contrastive Learning (SeCo). SeCo shows superior results under the linear protocol on action recognition (Kinetics), untrimmed activity recognition (ActivityNet) and object tracking (OTB-100). More remarkably, SeCo demonstrates considerable improvements over recent unsupervised pre-training techniques, and leads the accuracy by 2.96% and 6.47% against fully-supervised ImageNet pre-training in action recognition task on UCF101 and HMDB51, respectively. Source code is available at urlhttps://github.com/YihengZhang-CV/SeCo-Sequence-Contrastive-Learning.