# Approximation and learning with compositional tensor trains

arXiv: https://arxiv.org/abs/2512.18059 · HTML: https://mlsanity.com/paper/2512.18059/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 25 of 25)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis](https://mlsanity.com/paper/1806.03884/) (2018)
- [OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport](https://mlsanity.com/paper/2006.00104/) (2020)
- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [Revisiting the Nystrom Method for Improved Large-Scale Machine Learning](https://mlsanity.com/paper/1303.1849/) (2013)
- [Position: Optimization in SciML Should Employ the Function Space Geometry](https://mlsanity.com/paper/2402.07318/) (2024)
- [Minimum width for universal approximation using ReLU networks on compact domain](https://mlsanity.com/paper/2309.10402/) (2023)
- [Deep composition of tensor-trains using squared inverse Rosenblatt transports](https://mlsanity.com/paper/2007.06968/) (2020)
- [Invariance Properties of the Natural Gradient in Overparametrised Systems](https://mlsanity.com/paper/2206.15273/) (2022)
- [A modified MSA for stochastic control problems](https://mlsanity.com/paper/2007.05209/) (2020)
- [Tensor Random Projection for Low Memory Dimension Reduction](https://mlsanity.com/paper/2105.00105/) (2021)
- [Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part I](https://mlsanity.com/paper/2007.00118/) (2020)
- [Learning with tree tensor networks: complexity estimates and model selection](https://mlsanity.com/paper/2007.01165/) (2020)
- [Learning with tree-based tensor formats](https://mlsanity.com/paper/1811.04455/) (2018)
- [Approximation Theory of Tree Tensor Networks: Tensorized Multivariate Functions](https://mlsanity.com/paper/2101.11932/) (2021)
- [Generative Modelling with Tensor Train approximations of Hamilton--Jacobi--Bellman equations](https://mlsanity.com/paper/2402.15285/) (2024)
- [Learning high-dimensional probability distributions using tree tensor networks](https://mlsanity.com/paper/1912.07913/) (2019)
- [Tensor rank reduction via coordinate flows](https://mlsanity.com/paper/2207.11955/) (2022)
- [Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra](https://mlsanity.com/paper/2505.11638/) (2025)

## Built on by (most cited first, 1 of 1)

- [Natural Riemannian gradient for learning functional tensor networks](https://mlsanity.com/paper/2604.09263/) (2026)

## Abstract (arXiv)

We introduce compositional tensor trains (CTTs) for the approximation of multivariate functions, a class of models obtained by composing low-rank functions in the tensor-train format. This format can encode standard approximation tools, such as (sparse) polynomials, deep neural networks (DNNs) with fixed width, or tensor networks with arbitrary permutation of the inputs, or more general affine coordinate transformations, with similar complexities. This format can be viewed as a DNN with width exponential in the input dimension and structured weights matrices. Compared to DNNs, this format enables controlled compression at the layer level using efficient tensor algebra. On the optimization side, we derive a layerwise algorithm inspired by natural gradient descent, allowing to exploit efficient low-rank tensor algebra. This relies on low-rank estimations of Gram matrices, and tensor structured random sketching. Viewing the format as a discrete dynamical system, we also derive an optimization algorithm inspired by numerical methods in optimal control. Numerical experiments on regression tasks demonstrate the expressivity of the new format and the relevance of the proposed optimization algorithms. Overall, CTTs combine the expressivity of compositional models with the algorithmic efficiency of tensor algebra, offering a scalable alternative to standard deep neural networks.