# Learning to Linearize Under Uncertainty

arXiv: https://arxiv.org/abs/1506.03011 · HTML: https://mlsanity.com/paper/1506.03011/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2015.

## Built on by (most cited first, 25 of 25)

- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [Scaling 4D Representations](https://mlsanity.com/paper/2412.15212/) (2024)
- [Symmetry-Based Representations for Artificial and Biological General Intelligence](https://mlsanity.com/paper/2203.09250/) (2022)
- [Back to the Manifold: Recovering from Out-of-Distribution States](https://mlsanity.com/paper/2207.08673/) (2022)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference](https://mlsanity.com/paper/2403.04082/) (2024)
- [Localization with Sampling-Argmax](https://mlsanity.com/paper/2110.08825/) (2021)
- [Unsupervised Multi-object Segmentation Using Attention and Soft-argmax](https://mlsanity.com/paper/2205.13271/) (2022)
- [Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening](https://mlsanity.com/paper/2509.08502/) (2025)
- [PDWN: Pyramid Deformable Warping Network for Video Interpolation](https://mlsanity.com/paper/2104.01517/) (2021)
- [Learning predictable and robust neural representations by straightening image sequences](https://mlsanity.com/paper/2411.01777/) (2024)
- [Two-stage Rule-induction Visual Reasoning on RPMs with an Application to Video Prediction](https://mlsanity.com/paper/2111.12301/) (2021)
- [Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization](https://mlsanity.com/paper/2403.14973/) (2024)
- [Minimalistic Unsupervised Learning with the Sparse Manifold Transform](https://mlsanity.com/paper/2209.15261/) (2022)
- [A polar prediction model for learning to represent visual transformations](https://mlsanity.com/paper/2303.03432/) (2023)
- [GEOPHYS: The Geometry of Physical Plausibility](https://mlsanity.com/paper/2606.20707/) (2026)
- [Long Range Object-Level Monocular Depth Estimation for UAVs](https://mlsanity.com/paper/2302.08943/) (2023)
- [Prior based Sampling for Adaptive LiDAR](https://mlsanity.com/paper/2304.07099/) (2023)
- [Brain-like representational straightening of natural movies in robust feedforward neural networks](https://mlsanity.com/paper/2308.13870/) (2023)
- [Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling](https://mlsanity.com/paper/2504.05537/) (2025)
- [SAM 2++: Tracking Anything at Any Granularity](https://mlsanity.com/paper/2510.18822/) (2025)
- [Video Prediction Models as General Visual Encoders](https://mlsanity.com/paper/2405.16382/) (2024)
- [Nonlinear Model Order Reduction of Dynamical Systems in Process Engineering: Review and Comparison](https://mlsanity.com/paper/2506.12819/) (2025)

## Abstract (arXiv)

Training deep feature hierarchies to solve supervised learning tasks has achieved state of the art performance on many problems in computer vision. However, a principled way in which to train such hierarchies in the unsupervised setting has remained elusive. In this work we suggest a new architecture and loss for training deep feature hierarchies that linearize the transformations observed in unlabeled natural video sequences. This is done by training a generative model to predict video frames. We also address the problem of inherent uncertainty in prediction by introducing latent variables that are non-deterministic functions of the input into the network architecture.