# Learning Internal Representations of 3D Transformations from 2D Projected Inputs

arXiv: https://arxiv.org/abs/2303.17776 · HTML: https://mlsanity.com/paper/2303.17776/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 71 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 13 of 13)

- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.03650/) (2016)
- [Learning the Irreducible Representations of Commutative Lie Groups](https://mlsanity.com/paper/1402.4437/) (2014)
- [Variational Autoencoder with Learned Latent Structure](https://mlsanity.com/paper/2006.10597/) (2020)
- [Deep Non-Rigid Structure from Motion](https://mlsanity.com/paper/1908.00052/) (2019)
- [Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentation](https://mlsanity.com/paper/1510.02795/) (2015)
- [Representing Closed Transformation Paths in Encoded Network Latent Space](https://mlsanity.com/paper/1912.02644/) (2019)
- [Learning Identity-Preserving Transformations on Data Manifolds](https://mlsanity.com/paper/2106.12096/) (2021)

## Built on by (most cited first, 2 of 2)

- [On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory](https://mlsanity.com/paper/2412.11521/) (2024)
- [Latent Equivariant Operators for Robust Object Recognition: Promises and Challenges](https://mlsanity.com/paper/2602.18406/) (2026)

## Abstract (arXiv)

When interacting in a three dimensional world, humans must estimate 3D structure from visual inputs projected down to two dimensional retinal images. It has been shown that humans use the persistence of object shape over motion-induced transformations as a cue to resolve depth ambiguity when solving this underconstrained problem. With the aim of understanding how biological vision systems may internally represent 3D transformations, we propose a computational model, based on a generative manifold model, which can be used to infer 3D structure from the motion of 2D points. Our model can also learn representations of the transformations with minimal supervision, providing a proof of concept for how humans may develop internal representations on a developmental or evolutionary time scale. Focused on rotational motion, we show how our model infers depth from moving 2D projected points, learns 3D rotational transformations from 2D training stimuli, and compares to human performance on psychophysical structure-from-motion experiments.