# Adaptive and Implicit Regularization for Matrix Completion

arXiv: https://arxiv.org/abs/2208.05640 · HTML: https://mlsanity.com/paper/2208.05640/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [Dynamics of Deep Neural Networks and Neural Tangent Hierarchy](https://mlsanity.com/paper/1909.08156/) (2019)
- [Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks](https://mlsanity.com/paper/1704.06803/) (2017)
- [Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy](https://mlsanity.com/paper/2007.06738/) (2020)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Deep K-SVD Denoising](https://mlsanity.com/paper/1909.13164/) (2019)
- [Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization](https://mlsanity.com/paper/2006.08857/) (2020)
- [Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent](https://mlsanity.com/paper/2002.10583/) (2020)
- [MomentumRNN: Integrating Momentum into Recurrent Neural Networks](https://mlsanity.com/paper/2006.06919/) (2020)
- [Algorithms, Initializations, and Convergence for the Nonnegative Matrix Factorization](https://mlsanity.com/paper/1407.7299/) (2014)
- [Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum](https://mlsanity.com/paper/2012.02188/) (2020)
- [Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization](https://mlsanity.com/paper/2110.09057/) (2021)
- [Learned Greedy Method (LGM): A Novel Neural Architecture for Sparse Coding and Beyond](https://mlsanity.com/paper/2010.07069/) (2020)
- [A regularized deep matrix factorized model of matrix completion for image restoration](https://mlsanity.com/paper/2007.14581/) (2020)
- [Spectral Geometric Matrix Completion](https://mlsanity.com/paper/1911.07255/) (2019)

## Built on by (most cited first, 5 of 5)

- [Regularize implicit neural representation by itself](https://mlsanity.com/paper/2303.15484/) (2023)
- [NeurTV: Total Variation on the Neural Domain](https://mlsanity.com/paper/2405.17241/) (2024)
- [H2TF for Hyperspectral Image Denoising: Where Hierarchical Nonlinear Transform Meets Hierarchical Matrix Factorization](https://mlsanity.com/paper/2304.11141/) (2023)
- [DTR: A Unified Deep Tensor Representation Framework for Multimedia Data Recovery](https://mlsanity.com/paper/2407.05267/) (2024)
- [Long-Term Prediction Accuracy Improvement of Data-Driven Medium-Range Global Weather Forecast](https://mlsanity.com/paper/2407.01598/) (2024)

## Abstract (arXiv)

The explicit low-rank regularization, e.g., nuclear norm regularization, has been widely used in imaging sciences. However, it has been found that implicit regularization outperforms explicit ones in various image processing tasks. Another issue is that the fixed explicit regularization limits the applicability to broad images since different images favor different features captured by different explicit regularizations. As such, this paper proposes a new adaptive and implicit low-rank regularization that captures the low-rank prior dynamically from the training data. The core of our new adaptive and implicit low-rank regularization is parameterizing the Laplacian matrix in the Dirichlet energy-based regularization, which we call the regularization AIR. Theoretically, we show that the adaptive regularization of ReTwoAIR enhances the implicit regularization and vanishes at the end of training. We validate AIR's effectiveness on various benchmark tasks, indicating that the AIR is particularly favorable for the scenarios when the missing entries are non-uniform. The code can be found at https://github.com/lizhemin15/AIR-Net.