# Who Said Neural Networks Aren't Linear?

arXiv: https://arxiv.org/abs/2510.08570 · HTML: https://mlsanity.com/paper/2510.08570/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [VAMPnets: Deep learning of molecular kinetics](https://mlsanity.com/paper/1710.06012/) (2017)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons](https://mlsanity.com/paper/1305.2982/) (2013)
- [Idempotent Generative Network](https://mlsanity.com/paper/2311.01462/) (2023)
- [Learning Group Structure and Disentangled Representations of Dynamical Environments](https://mlsanity.com/paper/2002.06991/) (2020)
- [Representation Learning via Manifold Flattening and Reconstruction](https://mlsanity.com/paper/2305.01777/) (2023)
- [One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling](https://mlsanity.com/paper/2505.13358/) (2025)
- [IT3: Idempotent Test-Time Training](https://mlsanity.com/paper/2410.04201/) (2024)

## Built on by (most cited first, 2 of 2)

- [Pseudo-Invertible Neural Networks](https://mlsanity.com/paper/2602.06042/) (2026)
- [The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment](https://mlsanity.com/paper/2604.06377/) (2026)

## Abstract (arXiv)

Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, f:X → Y. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a linear operator A between two invertible neural networks, f(x)=gy-1(A gx(x)), the corresponding vector spaces X and Y are induced by newly defined addition and scaling actions derived from gx and gy. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce idempotency (i.e. f(f(x))=f(x)) on networks leading to a globally projective generative model and to demonstrate modular style transfer.