Best rank-k approximations for tensors: generalizing Eckart-Young

Given a tensor f in a Euclidean tensor space, we are interested in the critical points of the distance function from f to the set of tensors of rank at most k, which we call the critical rank-at-most-k tensors for f. When f is a matrix, the critical rank-one matrices for f correspond to the singular pairs of f. The critical rank-one tensors for f lie in a linear subspace Hf, the critical space of f. Our main result is that, for any k, the critical rank-at-most-k tensors for a sufficiently general f also lie in the critical space Hf. This is the part of Eckart-Young Theorem that generalizes from matrices to tensors. Moreover, we show that when the tensor format satisfies the triangle inequalities, the critical space Hf is spanned by the complex critical rank-one tensors. Since f itself belongs to Hf, we deduce that also f itself is a linear combination of its critical rank-one tensors.