# Unsupervised Learning of Spatiotemporally Coherent Metrics

arXiv: https://arxiv.org/abs/1412.6056 · HTML: https://mlsanity.com/paper/1412.6056/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2014.

## Built on by (most cited first, 30 of 46)

- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective](https://mlsanity.com/paper/2103.17263/) (2021)
- [Siamese Masked Autoencoders](https://mlsanity.com/paper/2305.14344/) (2023)
- [Diffusion Models as Masked Autoencoders](https://mlsanity.com/paper/2304.03283/) (2023)
- [Self-supervised Video Transformer](https://mlsanity.com/paper/2112.01514/) (2021)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Learning by Aligning Videos in Time](https://mlsanity.com/paper/2103.17260/) (2021)
- [Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning](https://mlsanity.com/paper/2303.16535/) (2023)
- [DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control](https://mlsanity.com/paper/2409.12192/) (2024)
- [Long-Short Temporal Contrastive Learning of Video Transformers](https://mlsanity.com/paper/2106.09212/) (2021)
- [ReSkin: versatile, replaceable, lasting tactile skins](https://mlsanity.com/paper/2111.00071/) (2021)
- [Noether Networks: Meta-Learning Useful Conserved Quantities](https://mlsanity.com/paper/2112.03321/) (2021)
- [Learning to Align Sequential Actions in the Wild](https://mlsanity.com/paper/2111.09301/) (2021)
- [Self-supervised video pretraining yields robust and more human-aligned visual representations](https://mlsanity.com/paper/2210.06433/) (2022)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning](https://mlsanity.com/paper/2106.04480/) (2021)
- [TempCLR: Temporal Alignment Representation with Contrastive Learning](https://mlsanity.com/paper/2212.13738/) (2022)
- [Temporal Coherent Test-Time Optimization for Robust Video Classification](https://mlsanity.com/paper/2302.14309/) (2023)
- [MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior](https://mlsanity.com/paper/2207.10553/) (2022)
- [HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2301.03330/) (2023)
- [Joint Self-Supervised Video Alignment and Action Segmentation](https://mlsanity.com/paper/2503.16832/) (2025)
- [Language-based Action Concept Spaces Improve Video Self-Supervised Learning](https://mlsanity.com/paper/2307.10922/) (2023)
- [Leveraging Time Irreversibility with Order-Contrastive Pre-training](https://mlsanity.com/paper/2111.02599/) (2021)
- [Palm up: Playing in the Latent Manifold for Unsupervised Pretraining](https://mlsanity.com/paper/2210.10913/) (2022)

## Abstract (arXiv)

Current state-of-the-art classification and detection algorithms rely on supervised training. In this work we study unsupervised feature learning in the context of temporally coherent video data. We focus on feature learning from unlabeled video data, using the assumption that adjacent video frames contain semantically similar information. This assumption is exploited to train a convolutional pooling auto-encoder regularized by slowness and sparsity. We establish a connection between slow feature learning to metric learning and show that the trained encoder can be used to define a more temporally and semantically coherent metric.