Precise Performance of Linear Denoisers in the Proportional Regime

In the present paper we study the performance of linear denoisers for noisy data of the form x + z, where x ∈ ℝd is the desired data with zero mean and unknown covariance Sigma, and z ~ N(0, Sigmaz) is additive noise. Since the covariance Sigma is not known, the standard Wiener filter cannot be employed for denoising. Instead we assume we are given samples x1,dots,xn ∈ ℝd from the true distribution. A standard approach would then be to estimate Sigma from the samples and use it to construct an ``empirical" Wiener filter. However, in this paper, motivated by the denoising step in diffusion models, we take a different approach whereby we train a linear denoiser W from the data itself. In particular, we synthetically construct noisy samples hatxi of the data by injecting the samples with Gaussian noise with covariance Sigma1 neq Sigmaz and find the best W that approximates Whatxi ≈ xi in a least-squares sense. In the proportional regime fracnd rightarrow kappa > 1 we use the it Convex Gaussian Min-Max Theorem (CGMT) to analytically find the closed form expression for the generalization error of the denoiser obtained from this process. Using this expression one can optimize over Sigma1 to find the best possible denoiser. Our numerical simulations show that our denoiser outperforms the ``empirical" Wiener filter in many scenarios and approaches the optimal Wiener filter as kapparightarrow∞.