{"list":"other","total":56,"rows":[{"id":"1912.01703","title":"PyTorch: An Imperative Style, High-Performance Deep Learning Library","created":"2019-12-03","cited_by_count":20305},{"id":"2010.01412","title":"Sharpness-Aware Minimization for Efficiently Improving Generalization","created":"2020-10-03","cited_by_count":1417},{"id":"1609.04836","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","created":"2016-09-15","cited_by_count":1352},{"id":"1712.09913","title":"Visualizing the Loss Landscape of Neural Nets","created":"2017-12-28","cited_by_count":1130},{"id":"1309.5549","title":"Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming","created":"2013-09-22","cited_by_count":724},{"id":"1912.02178","title":"Fantastic Generalization Measures and Where to Find Them","created":"2019-12-04","cited_by_count":513},{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427},{"id":"1804.08838","title":"Measuring the Intrinsic Dimension of Objective Landscapes","created":"2018-04-24","cited_by_count":370},{"id":"1901.10159","title":"An Investigation into Neural Net Optimization via Hessian Eigenvalue Density","created":"2019-01-29","cited_by_count":292},{"id":"1706.04454","title":"Empirical Analysis of the Hessian of Over-Parametrized Neural Networks","created":"2017-06-14","cited_by_count":290},{"id":"1812.04754","title":"Gradient Descent Happens in a Tiny Subspace","created":"2018-12-12","cited_by_count":217},{"id":"1611.07476","title":"Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond","created":"2016-11-22","cited_by_count":159},{"id":"2209.15594","title":"Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability","created":"2022-09-30","cited_by_count":155},{"id":"2002.09572","title":"The Break-Even Point on Optimization Trajectories of Deep Neural Networks","created":"2020-02-21","cited_by_count":150},{"id":"2206.06232","title":"Towards Understanding Sharpness-Aware Minimization","created":"2022-06-13","cited_by_count":146},{"id":"2002.03305","title":"Momentum Improves Normalized SGD","created":"2020-02-09","cited_by_count":139},{"id":"2006.07322","title":"Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks","created":"2020-06-12","cited_by_count":138},{"id":"2205.09745","title":"Understanding Gradient Descent on Edge of Stability in Deep Learning","created":"2022-05-19","cited_by_count":128},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"1904.09080","title":"Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process","created":"2019-04-19","cited_by_count":123},{"id":"2106.06530","title":"Label Noise SGD Provably Prefers Flat Global Minimizers","created":"2021-06-11","cited_by_count":121},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2110.06914","title":"What Happens after SGD Reaches Zero Loss? --A Mathematical Framework","created":"2021-10-13","cited_by_count":107},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"2008.11865","title":"Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra","created":"2020-08-27","cited_by_count":94},{"id":"1902.00744","title":"Asymmetric Valleys: Beyond Sharp and Flat Local Minima","created":"2019-02-02","cited_by_count":90},{"id":"2007.03813","title":"Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification","created":"2020-07-07","cited_by_count":90},{"id":"1807.05031","title":"On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length","created":"2018-07-13","cited_by_count":88},{"id":"2204.01050","title":"Understanding the unstable convergence of gradient descent","created":"2022-04-03","cited_by_count":86},{"id":"2206.07085","title":"Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction","created":"2022-06-14","cited_by_count":86},{"id":"2208.11195","title":"Robustness to Unbounded Smoothness of Generalized SignSGD","created":"2022-08-23","cited_by_count":81},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"1901.08244","title":"Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians","created":"2019-01-24","cited_by_count":68},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58},{"id":"2210.03294","title":"Understanding Edge-of-Stability Training Dynamics with a Minimalist Example","created":"2022-10-07","cited_by_count":56},{"id":"2210.01513","title":"The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minima","created":"2022-10-04","cited_by_count":51},{"id":"2212.07469","title":"Learning threshold neurons via the \"edge of stability\"","created":"2022-12-14","cited_by_count":51},{"id":"1803.05591","title":"On the insufficiency of existing momentum schemes for Stochastic Optimization","created":"2018-03-15","cited_by_count":49},{"id":"2305.11788","title":"Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability","created":"2023-05-19","cited_by_count":42},{"id":"2306.04815","title":"Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning","created":"2023-06-07","cited_by_count":31},{"id":"2305.15287","title":"The Crucial Role of Normalization in Sharpness-Aware Minimization","created":"2023-05-24","cited_by_count":26},{"id":"2307.04204","title":"Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory","created":"2023-07-09","cited_by_count":26},{"id":"2402.01567","title":"Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise","created":"2024-02-02","cited_by_count":26},{"id":"2302.08692","title":"SAM operates far from home: eigenvalue regularization as a dynamical phenomenon","created":"2023-02-17","cited_by_count":24},{"id":"2305.13064","title":"Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond","created":"2023-05-22","cited_by_count":21},{"id":"2306.09850","title":"Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima","created":"2023-06-16","cited_by_count":20},{"id":"2011.04720","title":"Improving Neural Network Training in Low Dimensional Random Bases","created":"2020-11-09","cited_by_count":19},{"id":"2306.09000","title":"When and Why Momentum Accelerates SGD:An Empirical Study","created":"2023-06-15","cited_by_count":19},{"id":"2307.15196","title":"The Marginal Value of Momentum for Small Learning Rate SGD","created":"2023-07-27","cited_by_count":18},{"id":"2404.19261","title":"High dimensional analysis reveals conservative sharpening and a stochastic edge of stability","created":"2024-04-30","cited_by_count":18},{"id":"2309.12488","title":"Sharpness-Aware Minimization and the Edge of Stability","created":"2023-09-21","cited_by_count":17},{"id":"2106.00001","title":"Privately Learning Subspaces","created":"2021-05-28","cited_by_count":16},{"id":"2305.15659","title":"How to escape sharp minima with random perturbations","created":"2023-05-25","cited_by_count":13},{"id":"2310.03010","title":"Spectral alignment of stochastic gradient descent for high-dimensional classification tasks","created":"2023-10-04","cited_by_count":13},{"id":"2206.03483","title":"Few-Shot Learning by Dimensionality Reduction in Gradient Space","created":"2022-06-07","cited_by_count":8},{"id":"2401.06604","title":"Identifying Policy Gradient Subspaces","created":"2024-01-12","cited_by_count":5}],"next_cursor":null}