Near Optimal Sketching of Low-Rank Tensor Regression

We study the least squares regression problem beginalign* min_Theta ∈ Sodot D,R |ATheta-b|2, endalign* where Sodot D,R is the set of Theta for which Theta = sumr=1R θ1(r) circ cdots circ θD(r) for vectors θd(r) ∈ ℝpd for all r ∈ [R] and d ∈ [D], and circ denotes the outer product of vectors. That is, Theta is a low-dimensional, low-rank tensor. This is motivated by the fact that the number of parameters in Theta is only R · sumd=1D pd, which is significantly smaller than the prodd=1D pd number of parameters in ordinary least squares regression. We consider the above CP decomposition model of tensors Theta, as well as the Tucker decomposition. For both models we show how to apply data dimensionality reduction techniques based on it sparse random projections Phi ∈ ℝm × n, with m ll n, to reduce the problem to a much smaller problem minTheta |Phi A Theta - Phi b|2, for which if Theta' is a near-optimum to the smaller problem, then it is also a near optimum to the original problem. We obtain significantly smaller dimension and sparsity in Phi than is possible for ordinary least squares regression, and we also provide a number of numerical simulations supporting our theory.