Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods
Not on the map. Why?
Abstract (arXiv)
Autoencoders are a popular model in many branches of machine learning and lossy data compression. However, their fundamental limits, the performance of gradient methods and the features learnt during optimization remain poorly understood, even in the two-layer setting. In fact, earlier work has considered either linear autoencoders or specific training regimes (leading to vanishing or diverging compression rates). Our paper addresses this gap by focusing on non-linear two-layer autoencoders trained in the challenging proportional regime in which the input dimension scales linearly with the size of the representation. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods; their structure is also unveiled, thus leading to a concise description of the features obtained via training. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders. Finally, while the results are proved for Gaussian data, numerical simulations on standard datasets display the universality of the theoretical predictions.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 76 references that are in the map. First posted on arXiv 27 Dec 2022.
Builds on
36 of 76 references.
References · 36
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2021Convex optimizationcited by 2,515
- 2016End-to-end Optimized Image Compressioncited by 696
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2020Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modulescited by 590
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
- 2017Lossy Image Compression with Compressive Autoencoderscited by 294
- 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
- 2015Coordinate Descent Algorithmscited by 219
- 2013Expectation Propagation for approximate Bayesian inferencecited by 174
- 2022Lossy Image Compression with Conditional Diffusion Modelscited by 170
- 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 152
24 more
- 2020Nonlinear Transform Codingcited by 136
- 2018Recent Advances in Autoencoder-Based Representation Learningcited by 133
- 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
- 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 125
- 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
- 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
- 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
- 2022Lossy Compression with Gaussian Diffusioncited by 90
- 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
- 2022An Introduction to Neural Data Compressioncited by 79
- 2022Universality of empirical risk minimizationcited by 75
- 2019Loss Landscapes of Regularized Linear Autoencoderscited by 46
- 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
- 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
- 2020Regularized linear autoencoders recover the principal components, eventuallycited by 26
- 2021Analysis of feature learning in weight-tied autoencoders via the mean field lenscited by 14
- 2021Towards Empirical Sandwich Bounds on the Rate-Distortion Functioncited by 14
- 2017Inference in Deep Networks in High Dimensionscited by 13
- 2019Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysiscited by 13
- 2017Sparse Coding and Autoencoderscited by 11
- 2020Neural Networks Optimally Compress the Sawbridgecited by 11
- 2022Binary Iterative Hard Thresholding Converges with Optimal Number of Measurements for 1-Bit Compressed Sensingcited by 9
- 2016Expectation Consistent Approximate Inference: Generalizations and Convergencecited by 6
- 2021A Mechanism for Producing Aligned Latent Spaces with Autoencoderscited by 5
Built on by
Citing papers per year: 2023: 1; 2024: 2.
3 papers.