# Shuffle and Learn: Unsupervised Learning using Temporal Order Verification

arXiv: https://arxiv.org/abs/1603.08561 · HTML: https://mlsanity.com/paper/1603.08561/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2016.

## Built on by (most cited first, 13 of 13)

- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models](https://mlsanity.com/paper/2410.23266/) (2024)
- [Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity](https://mlsanity.com/paper/2112.05883/) (2021)
- [CDFSL-V: Cross-Domain Few-Shot Learning for Videos](https://mlsanity.com/paper/2309.03989/) (2023)
- [A Large-Scale Analysis on Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2306.06010/) (2023)
- [Traffic Scenario Clustering by Iterative Optimisation of Self-Supervised Networks Using a Random Forest Activation Pattern Similarity](https://mlsanity.com/paper/2105.07639/) (2021)
- [Leveraging Self-Supervised Training for Unintentional Action Recognition](https://mlsanity.com/paper/2209.11870/) (2022)
- [A Large-Scale Analysis on Contextual Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2504.06153/) (2025)
- [C-LEAD: Contrastive Learning for Enhanced Adversarial Defense](https://mlsanity.com/paper/2510.27249/) (2025)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)
- [HomE: Homography-Equivariant Video Representation Learning](https://mlsanity.com/paper/2306.01623/) (2023)
- [MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning](https://mlsanity.com/paper/2306.02252/) (2023)
- [Learning to Anticipate Future with Dynamic Context Removal](https://mlsanity.com/paper/2204.02587/) (2022)

## Abstract (arXiv)

In this paper, we present an approach for learning a visual representation from the raw spatiotemporal signals in videos. Our representation is learned without supervision from semantic labels. We formulate our method as an unsupervised sequential verification task, i.e., we determine whether a sequence of frames from a video is in the correct temporal order. With this simple task and no semantic labels, we learn a powerful visual representation using a Convolutional Neural Network (CNN). The representation contains complementary information to that learned from supervised image datasets like ImageNet. Qualitative results show that our method captures information that is temporally varying, such as human pose. When used as pre-training for action recognition, our method gives significant gains over learning without external data on benchmark datasets like UCF101 and HMDB51. To demonstrate its sensitivity to human pose, we show results for pose estimation on the FLIC and MPII datasets that are competitive, or better than approaches using significantly more supervision. Our method can be combined with supervised representations to provide an additional boost in accuracy.