# Representation Learning via Manifold Flattening and Reconstruction

arXiv: https://arxiv.org/abs/2305.01777 · HTML: https://mlsanity.com/paper/2305.01777/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 91 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions](https://mlsanity.com/paper/2209.11215/) (2022)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform](https://mlsanity.com/paper/2002.01113/) (2020)
- [On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence](https://mlsanity.com/paper/2207.04630/) (2022)
- [Intrinsic Isometric Manifold Learning with Application to Localization](https://mlsanity.com/paper/1806.00556/) (2018)
- [Scalable Out-of-Sample Extension of Graph Embeddings Using Deep Neural Networks](https://mlsanity.com/paper/1508.04422/) (2015)
- [Random projections of random manifolds](https://mlsanity.com/paper/1607.04331/) (2016)

## Built on by (most cited first, 11 of 11)

- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [Action-Minimization Meets Generative Modeling: Efficient Transition Path Sampling with the Onsager-Machlup Functional](https://mlsanity.com/paper/2504.18506/) (2025)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Masked Completion via Structured Diffusion with White-Box Transformers](https://mlsanity.com/paper/2404.02446/) (2024)
- [Words in Motion: Extracting Interpretable Control Vectors for Motion Transformers](https://mlsanity.com/paper/2406.11624/) (2024)
- [LightFF: Lightweight Inference for Forward-Forward Algorithm](https://mlsanity.com/paper/2404.05241/) (2024)
- [Who Said Neural Networks Aren't Linear?](https://mlsanity.com/paper/2510.08570/) (2025)
- [Latent Manifold Reconstruction and Representation with Topological and Geometrical Regularization](https://mlsanity.com/paper/2505.04412/) (2025)
- [Riemannian Archetypal Analysis: Interpretable non-linear data analysis on deformed star distributions](https://mlsanity.com/paper/2605.24113/) (2026)
- [Normalized Space Alignment: A Versatile Metric for Representation Analysis](https://mlsanity.com/paper/2411.04512/) (2024)
- [Superposed Latent Autoencoder](https://mlsanity.com/paper/2609.01158/) (2026)

## Abstract (arXiv)

This work proposes an algorithm for explicitly constructing a pair of neural networks that linearize and reconstruct an embedded submanifold, from finite samples of this manifold. Our such-generated neural networks, called Flattening Networks (FlatNet), are theoretically interpretable, computationally feasible at scale, and generalize well to test data, a balance not typically found in manifold-based learning methods. We present empirical results and comparisons to other models on synthetic high-dimensional manifold data and 2D image data. Our code is publicly available.