# Tensors, Learning, and 'Kolmogorov Extension' for Finite-alphabet Random Vectors

arXiv: https://arxiv.org/abs/1712.00205 · HTML: https://mlsanity.com/paper/1712.00205/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2017.

## Built on by (most cited first, 16 of 16)

- [Tensor-Train Density Estimation](https://mlsanity.com/paper/2108.00089/) (2021)
- [Beyond Smoothness: Incorporating Low-Rank Analysis into Nonparametric Density Estimation](https://mlsanity.com/paper/2204.00930/) (2022)
- [Stochastic Mirror Descent for Low-Rank Tensor Decomposition Under Non-Euclidean Losses](https://mlsanity.com/paper/2104.14562/) (2021)
- [Recovery of Joint Probability Distribution from one-way marginals: Low rank Tensors and Random Projections](https://mlsanity.com/paper/2103.11864/) (2021)
- [Learning From Crowdsourced Noisy Labels: A Signal Processing Perspective](https://mlsanity.com/paper/2407.06902/) (2024)
- [Generalized Identifiability Bounds for Mixture Models with Grouped Samples](https://mlsanity.com/paper/2207.11164/) (2022)
- [Minimizing low-rank models of high-order tensors: Hardness, span, tight relaxation, and applications](https://mlsanity.com/paper/2210.11413/) (2022)
- [TERM Model: Tensor Ring Mixture Model for Density Estimation](https://mlsanity.com/paper/2312.08075/) (2023)
- [Low-Rank Tensors for Multi-Dimensional Markov Models](https://mlsanity.com/paper/2411.02098/) (2024)
- [Joint Probability Estimation Using Tensor Decomposition and Dictionaries](https://mlsanity.com/paper/2203.01667/) (2022)
- [Tensor Decomposition based Personalized Federated Learning](https://mlsanity.com/paper/2208.12959/) (2022)
- [Estimating Joint Probability Distribution With Low-Rank Tensor Decomposition, Radon Transforms and Dictionaries](https://mlsanity.com/paper/2304.08740/) (2023)
- [Learning Multivariate CDFs and Copulas using Tensor Factorization](https://mlsanity.com/paper/2210.07132/) (2022)
- [Joint Bayesian Parameter and Model Order Estimation for Low-Rank Probability Mass Tensors](https://mlsanity.com/paper/2410.06329/) (2024)
- [A Unified MDL-based Binning and Tensor Factorization Framework for PDF Estimation](https://mlsanity.com/paper/2504.18686/) (2025)
- [Smooth Reparameterizations of Functions on Simplicial Product Spaces: Applications to Probabilistic Tensor Decomposition and Functional Data Registration](https://mlsanity.com/paper/2608.02576/) (2026)

## Abstract (arXiv)

Estimating the joint probability mass function (PMF) of a set of random variables lies at the heart of statistical learning and signal processing. Without structural assumptions, such as modeling the variables as a Markov chain, tree, or other graphical model, joint PMF estimation is often considered mission impossible - the number of unknowns grows exponentially with the number of variables. But who gives us the structural model? Is there a generic, `non-parametric' way to control joint PMF complexity without relying on a priori structural assumptions regarding the underlying probability model? Is it possible to discover the operational structure without biasing the analysis up front? What if we only observe random subsets of the variables, can we still reliably estimate the joint PMF of all? This paper shows, perhaps surprisingly, that if the joint PMF of any three variables can be estimated, then the joint PMF of all the variables can be provably recovered under relatively mild conditions. The result is reminiscent of Kolmogorov's extension theorem - consistent specification of lower-dimensional distributions induces a unique probability measure for the entire process. The difference is that for processes of limited complexity (rank of the high-dimensional PMF) it is possible to obtain complete characterization from only three-dimensional distributions. In fact not all three-dimensional PMFs are needed; and under more stringent conditions even two-dimensional will do. Exploiting multilinear algebra, this paper proves that such higher-dimensional PMF completion can be guaranteed - several pertinent identifiability results are derived. It also provides a practical and efficient algorithm to carry out the recovery task. Judiciously designed simulations and real-data experiments on movie recommendation and data classification are presented to showcase the effectiveness of the approach.