mlsanityData to 7 Oct 2026

Harnessing Orthogonality to Train Low-Rank Neural Networks

arXiv 2401.08505 ↗First posted 16 Jan 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

This study explores the learning dynamics of neural networks by analyzing the singular value decomposition (SVD) of their weights throughout training. Our investigation reveals that an orthogonal basis within each multidimensional weight's SVD representation stabilizes during training. Building upon this, we introduce Orthogonality-Informed Adaptive Low-Rank (OIALR) training, a novel training method exploiting the intrinsic orthogonality of neural networks. OIALR seamlessly integrates into existing training workflows with minimal accuracy loss, as demonstrated by benchmarking on various datasets and well-established network architectures. With appropriate hyperparameter tuning, OIALR can surpass conventional training setups, including those of state-of-the-art models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

34 of 52 references.

References · 34

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  9. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  10. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  11. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  12. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
22 more
  1. 2020Are we done with ImageNet?cited by 329
  2. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  3. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  4. 2019Structured Pruning of Large Language Modelscited by 201
  5. 2022Language model compression with weighted low-rank factorizationcited by 192
  6. 2020Directional convergence and alignment in deep learningcited by 172
  7. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  8. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  9. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  10. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  11. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  12. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  13. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  14. 2019Enhancing Explainability of Neural Networks through Architecture Constraintscited by 29
  15. 2018Trained Rank Pruning for Efficient Deep Neural Networkscited by 23
  16. 2019Play and Prune: Adaptive Filter Pruning for Deep Model Compressioncited by 15
  17. 2020Principal Component Networks: Parameter Reduction Early in Trainingcited by 9
  18. 2021Greenformer: Factorization Toolkit for Efficient Deep Neural Networkscited by 7
  19. 2023Massively Parallel Genetic Optimization through Asynchronous Propagation of Populationscited by 7
  20. 2022Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraintcited by 5
  21. 2020kDecay: Just adding k-decay items on Learning-Rate Schedule to improve Neural Networkscited by 4
  22. 2020Adaptive Low-Rank Factorization to regularize shallow and deep neural networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025A geometric framework for momentum-based optimizers for low-rank trainingcited by 5
  2. 2025On Information Geometry and Iterative Optimization in Model Compression: Operator Factorizationcited by 1
  3. 2024AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learningcited by 0
  4. 2026Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimizationcited by 0