# A polar prediction model for learning to represent visual transformations

arXiv: https://arxiv.org/abs/2303.03432 · HTML: https://mlsanity.com/paper/2303.03432/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 49 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Forecasting Sequential Data using Consistent Koopman Autoencoders](https://mlsanity.com/paper/2003.02236/) (2020)
- [Learning the Irreducible Representations of Commutative Lie Groups](https://mlsanity.com/paper/1402.4437/) (2014)
- [The Sparse Manifold Transform](https://mlsanity.com/paper/1806.08887/) (2018)
- [Learning to Linearize Under Uncertainty](https://mlsanity.com/paper/1506.03011/) (2015)
- [Predictive information in a sensory population](https://mlsanity.com/paper/1307.0225/) (2013)
- [Bispectral Neural Networks](https://mlsanity.com/paper/2209.03416/) (2022)
- [Disentangling images with Lie group transformations and sparse coding](https://mlsanity.com/paper/2012.12071/) (2020)
- [Maximum Entropy Models from Phase Harmonic Covariances](https://mlsanity.com/paper/1911.10017/) (2019)
- [WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction](https://mlsanity.com/paper/2211.14308/) (2022)

## Built on by (most cited first, 6 of 6)

- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Poisson Variational Autoencoder](https://mlsanity.com/paper/2405.14473/) (2024)
- [Learning predictable and robust neural representations by straightening image sequences](https://mlsanity.com/paper/2411.01777/) (2024)
- [Convergent Evolution: How Different Language Models Learn Similar Number Representations](https://mlsanity.com/paper/2604.20817/) (2026)
- [HieDG: A Hierarchical Discrete Geometry-Guided Framework for Multi-Animal Tracking](https://mlsanity.com/paper/2607.00494/) (2026)

## Abstract (arXiv)

All organisms make temporal predictions, and their evolutionary fitness level depends on the accuracy of these predictions. In the context of visual perception, the motions of both the observer and objects in the scene structure the dynamics of sensory signals, allowing for partial prediction of future signals based on past ones. Here, we propose a self-supervised representation-learning framework that extracts and exploits the regularities of natural videos to compute accurate predictions. We motivate the polar architecture by appealing to the Fourier shift theorem and its group-theoretic generalization, and we optimize its parameters on next-frame prediction. Through controlled experiments, we demonstrate that this approach can discover the representation of simple transformation groups acting in data. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation and rivals conventional deep networks, while maintaining interpretability and speed. Furthermore, the polar computations can be restructured into components resembling normalized simple and direction-selective complex cell models of primate V1 neurons. Thus, polar prediction offers a principled framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction.