# Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processing

arXiv: https://arxiv.org/abs/2203.06031 · HTML: https://mlsanity.com/paper/2203.06031/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [Proving the Lottery Ticket Hypothesis: Pruning is All You Need](https://mlsanity.com/paper/2002.00585/) (2020)
- [Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks](https://mlsanity.com/paper/1805.12076/) (2018)
- [Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds](https://mlsanity.com/paper/1605.07147/) (2016)
- [Tensor-Train Recurrent Neural Networks for Video Classification](https://mlsanity.com/paper/1707.01786/) (2017)
- [Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition](https://mlsanity.com/paper/2010.13309/) (2020)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)
- [Introduction to Tensor Decompositions and their Applications in Machine Learning](https://mlsanity.com/paper/1711.10781/) (2017)
- [A neural attention model for speech command recognition](https://mlsanity.com/paper/1808.08929/) (2018)
- [Long-term Forecasting using Higher Order Tensor RNNs](https://mlsanity.com/paper/1711.00073/) (2017)
- [On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression](https://mlsanity.com/paper/2008.07281/) (2020)
- [Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition](https://mlsanity.com/paper/1710.07324/) (2017)
- [Classical-to-Quantum Transfer Learning for Spoken Command Recognition Based on Quantum Neural Networks](https://mlsanity.com/paper/2110.08689/) (2021)
- [Generalized Max Pooling](https://mlsanity.com/paper/1406.0312/) (2014)
- [Analyzing Upper Bounds on Mean Absolute Errors for Deep Neural Network Based Vector-to-Vector Regression](https://mlsanity.com/paper/2008.05459/) (2020)
- [Tensor Decompositions in Deep Learning](https://mlsanity.com/paper/2002.11835/) (2020)
- [Listening to the World Improves Speech Command Recognition](https://mlsanity.com/paper/1710.08377/) (2017)
- [Exploring Deep Hybrid Tensor-to-Vector Network Architectures for Regression Based Speech Enhancement](https://mlsanity.com/paper/2007.13024/) (2020)
- [Exploiting Hybrid Models of Tensor-Train Networks for Spoken Command Recognition](https://mlsanity.com/paper/2201.10609/) (2022)

## Built on by (most cited first, 10 of 10)

- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Theoretical Error Performance Analysis for Variational Quantum Circuit Based Functional Regression](https://mlsanity.com/paper/2206.04804/) (2022)
- [Guaranteed Nonconvex Factorization Approach for Tensor Train Recovery](https://mlsanity.com/paper/2401.02592/) (2024)
- [TensorHyper-VQC: A Tensor-Train-Guided Hypernetwork for Robust and Scalable Variational Quantum Computing](https://mlsanity.com/paper/2508.01116/) (2025)
- [Quantum Machine Learning: An Interplay Between Quantum Computing and Machine Learning](https://mlsanity.com/paper/2411.09403/) (2024)
- [Pre-training Tensor-Train Networks Facilitates Machine Learning with Variational Quantum Circuits](https://mlsanity.com/paper/2306.03741/) (2023)
- [Leveraging Pre-Trained Neural Networks to Enhance Machine Learning with Variational Quantum Circuits](https://mlsanity.com/paper/2411.08552/) (2024)
- [Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation](https://mlsanity.com/paper/2506.16456/) (2025)
- [Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms](https://mlsanity.com/paper/2309.06577/) (2023)
- [Quantum LEGO Learning: A Modular Design Principle for Hybrid Artificial Intelligence](https://mlsanity.com/paper/2601.21780/) (2026)

## Abstract (arXiv)

This work focuses on designing low complexity hybrid tensor networks by considering trade-offs between the model complexity and practical performance. Firstly, we exploit a low-rank tensor-train deep neural network (TT-DNN) to build an end-to-end deep learning pipeline, namely LR-TT-DNN. Secondly, a hybrid model combining LR-TT-DNN with a convolutional neural network (CNN), which is denoted as CNN+(LR-TT-DNN), is set up to boost the performance. Instead of randomly assigning large TT-ranks for TT-DNN, we leverage Riemannian gradient descent to determine a TT-DNN associated with small TT-ranks. Furthermore, CNN+(LR-TT-DNN) consists of convolutional layers at the bottom for feature extraction and several TT layers at the top to solve regression and classification problems. We separately assess the LR-TT-DNN and CNN+(LR-TT-DNN) models on speech enhancement and spoken command recognition tasks. Our empirical evidence demonstrates that the LR-TT-DNN and CNN+(LR-TT-DNN) models with fewer model parameters can outperform the TT-DNN and CNN+(TT-DNN) counterparts.