mlsanityData to 14 Oct 2026

Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture

arXiv 2501.16322 ↗First posted 27 Jan 2025cs.LG math.OC stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Gradient descent for matrix factorization exhibits an implicit bias toward approximately low-rank solutions. While existing theories often assume the boundedness of iterates, empirically the bias persists even with unbounded sequences. This reflects a dynamic where factors develop low-rank structure while their magnitudes increase, tending to align with certain directions. To capture this behavior in a stable way, we introduce a new factorization model: X≈ UDV^top, where U and V are constrained within norm balls, while D is a diagonal factor allowing the model to span the entire search space. Experiments show that this model consistently exhibits a strong implicit bias, yielding truly (rather than approximately) low-rank solutions. Extending the idea to neural networks, we introduce a new model featuring constrained layers and diagonal components that achieves competitive performance on various regression and classification tasks while producing lightweight, low-rank representations.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 62 of its 106 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

62 of 106 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2020Knowledge Distillation: A Surveycited by 1,603
  7. 2020Designing Network Design Spacescited by 960
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  9. 2017Exploring Generalization in Deep Learningcited by 600
  10. 2020What is the State of Neural Network Pruning?cited by 583
  11. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  12. 2022MaxViT: Multi-Axis Vision Transformercited by 476
38 more
  1. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  2. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  3. 2017Implicit Regularization in Matrix Factorizationcited by 322
  4. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  5. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  6. 2021Physics-informed neural networks with hard constraints for inverse designcited by 280
  7. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  8. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  9. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  10. 2022Language model compression with weighted low-rank factorizationcited by 195
  11. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  13. 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 125
  14. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  15. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  16. 2015Constrained Convolutional Neural Networks for Weakly Supervised Segmentationcited by 100
  17. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  18. 2017Implicit Regularization in Deep Learningcited by 90
  19. 2016Group Sparse Regularization for Deep Neural Networkscited by 89
  20. 2014Guaranteed Matrix Completion via Non-convex Factorizationcited by 82
  21. 2018Constrained-CNN losses for weakly supervised segmentationcited by 62
  22. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 59
  23. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  24. 2017Constrained Deep Weak Supervision for Histopathology Image Segmentationcited by 51
  25. 2022Rank Diminishing in Deep Neural Networkscited by 47
  26. 2016The non-convex Burer-Monteiro approach works on smooth semidefinite programscited by 46
  27. 2017Imposing Hard Constraints on Deep Networks: Promises and Limitationscited by 43
  28. 2015Dropping Convexity for Faster Semi-definite Optimizationcited by 40
  29. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  30. 2016Non-square matrix sensing without spurious local minima via the Burer-Monteiro approachcited by 38
  31. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 34
  32. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  33. 2019An Inexact Augmented Lagrangian Framework for Nonconvex Optimization with Nonlinear Constraintscited by 29
  34. 2016Gradient Descent Converges to Minimizerscited by 25
  35. 2020Deep Neural Network Training with Frank-Wolfecited by 23
  36. 2016Finding Low-Rank Solutions via Non-Convex Matrix Factorization, Efficiently and Provablycited by 22
  37. 2018Lifted Neural Networkscited by 20
  38. 2018Deterministic guarantees for Burer-Monteiro factorizations of smooth semidefinite programscited by 19

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Sparsity Induced Identifiability in Matrix Tri-Factorisationcited by 0