# Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization

arXiv: https://arxiv.org/abs/2303.11003 · HTML: https://mlsanity.com/paper/2303.11003/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 94 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 30 of 76)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [SpeedNet: Learning the Speediness in Videos](https://mlsanity.com/paper/2004.06130/) (2020)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [Self-supervised Video Representation Learning by Pace Prediction](https://mlsanity.com/paper/2008.05861/) (2020)

## Built on by (most cited first, 12 of 12)

- [SIGMA: Sinkhorn-Guided Masked Video Modeling](https://mlsanity.com/paper/2407.15447/) (2024)
- [No More Shortcuts: Realizing the Potential of Temporal Self-Supervision](https://mlsanity.com/paper/2312.13008/) (2023)
- [Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening](https://mlsanity.com/paper/2509.08502/) (2025)
- [SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning](https://mlsanity.com/paper/2504.00527/) (2025)
- [LocoMotion: Learning Motion-Focused Video-Language Representations](https://mlsanity.com/paper/2410.12018/) (2024)
- [SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning](https://mlsanity.com/paper/2504.05706/) (2025)
- [Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning](https://mlsanity.com/paper/2606.07770/) (2026)
- [Sync from the Sea: Retrieving Alignable Videos from Large-Scale Datasets](https://mlsanity.com/paper/2409.01445/) (2024)
- [Structured-Noise Masked Modeling for Video, Audio and Beyond](https://mlsanity.com/paper/2503.16311/) (2025)
- [Advancing Video Self-Supervised Learning via Image Foundation Models](https://mlsanity.com/paper/2505.19218/) (2025)
- [Privacy Beyond Pixels: Latent Anonymization for Privacy-Preserving Video Understanding](https://mlsanity.com/paper/2511.08666/) (2025)
- [Self-Supervised Video Representation Learning in a Heuristic Decoupled Perspective](https://mlsanity.com/paper/2407.14069/) (2024)

## Abstract (arXiv)

We propose a self-supervised method for learning motion-focused video representations. Existing approaches minimize distances between temporally augmented videos, which maintain high spatial similarity. We instead propose to learn similarities between videos with identical local motion dynamics but an otherwise different appearance. We do so by adding synthetic motion trajectories to videos which we refer to as tubelets. By simulating different tubelet motions and applying transformations, such as scaling and rotation, we introduce motion patterns beyond what is present in the pretraining data. This allows us to learn a video representation that is remarkably data efficient: our approach maintains performance when using only 25% of the pretraining videos. Experiments on 10 diverse downstream settings demonstrate our competitive performance and generalizability to new domains and fine-grained actions.