mlsanityData to 7 Oct 2026

Spectral Tensor Train Parameterization of Deep Learning Layers

arXiv 2103.04217 ↗First posted 7 Mar 2021cs.LG cs.CV stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting and both compression and improved training stability in the generative adversarial training setting.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 7 Mar 2021.

Builds on

25 of 42 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2018Demystifying MMD GANscited by 1,268
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  12. 2015Tensorizing Neural Networkscited by 246
13 more
  1. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  2. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  3. 2016Mode Regularized Generative Adversarial Networkscited by 123
  4. 2018The Singular Values of Convolutional Layerscited by 95
  5. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  6. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  7. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  8. 2018Wide Compression: Tensor Ring Netscited by 50
  9. 2020Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interferencecited by 47
  10. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  11. 2018Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterizationcited by 35
  12. 2019Spectral Regularization for Combating Mode Collapse in GANscited by 31
  13. 2020T-Basis: a Compact Representation for Neural Networkscited by 21

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5.

6 papers.

  1. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  2. 2022ZippyPoint: Fast Interest Point Detection, Description, and Matching through Mixed Precision Discretizationcited by 10
  3. 2022Towards Practical Control of Singular Values of Convolutional Layerscited by 9
  4. 2022TT-NF: Tensor Train Neural Fieldscited by 7
  5. 2022Cost-efficient Gaussian Tensor Network Embeddings for Tensor-structured Inputscited by 5
  6. 2022Canonical convolutional neural networkscited by 3