Who Said Neural Networks Aren't Linear?

Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, f:X → Y. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a linear operator A between two invertible neural networks, f(x)=gy-1(A gx(x)), the corresponding vector spaces X and Y are induced by newly defined addition and scaling actions derived from gx and gy. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce idempotency (i.e. f(f(x))=f(x)) on networks leading to a globally projective generative model and to demonstrate modular style transfer.