{"list":"other","total":73,"rows":[{"id":"1412.6980","title":"Adam: A Method for Stochastic Optimization","created":"2014-12-22","cited_by_count":37240},{"id":"2010.11929","title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","created":"2020-10-22","cited_by_count":30186},{"id":"1502.01852","title":"Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification","created":"2015-02-06","cited_by_count":3611},{"id":"1806.07572","title":"Neural Tangent Kernel: Convergence and Generalization in Neural Networks","created":"2018-06-20","cited_by_count":2621},{"id":"2303.18223","title":"A Survey of Large Language Models","created":"2023-03-31","cited_by_count":2576},{"id":"1503.05671","title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","created":"2015-03-19","cited_by_count":796},{"id":"1902.06720","title":"Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent","created":"2019-02-18","cited_by_count":738},{"id":"1603.06560","title":"Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization","created":"2016-03-21","cited_by_count":699},{"id":"1812.07956","title":"On Lazy Training in Differentiable Programming","created":"2018-12-19","cited_by_count":675},{"id":"1711.00165","title":"Deep Neural Networks as Gaussian Processes","created":"2017-11-01","cited_by_count":622},{"id":"1904.11955","title":"On Exact Computation with an Infinitely Wide Neural Net","created":"2019-04-26","cited_by_count":573},{"id":"1412.1193","title":"New insights and perspectives on the natural gradient method","created":"2014-12-03","cited_by_count":443},{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427},{"id":"1805.09545","title":"On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport","created":"2018-05-24","cited_by_count":426},{"id":"1502.03492","title":"Gradient-based Hyperparameter Optimization through Reversible Learning","created":"2015-02-11","cited_by_count":360},{"id":"1901.10159","title":"An Investigation into Neural Net Optimization via Hessian Eigenvalue Density","created":"2019-01-29","cited_by_count":292},{"id":"1912.07145","title":"PyHessian: Neural Networks Through the Lens of the Hessian","created":"2019-12-16","cited_by_count":292},{"id":"1706.04454","title":"Empirical Analysis of the Hessian of Over-Parametrized Neural Networks","created":"2017-06-14","cited_by_count":290},{"id":"2203.03466","title":"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer","created":"2022-03-07","cited_by_count":288},{"id":"1502.05700","title":"Scalable Bayesian Optimization Using Deep Neural Networks","created":"2015-02-19","cited_by_count":284},{"id":"2002.04486","title":"Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss","created":"2020-02-11","cited_by_count":268},{"id":"1703.01785","title":"Forward and Reverse Gradient-Based Hyperparameter Optimization","created":"2017-03-06","cited_by_count":234},{"id":"1502.07943","title":"Non-stochastic Best Arm Identification and Hyperparameter Optimization","created":"2015-02-27","cited_by_count":221},{"id":"1812.04754","title":"Gradient Descent Happens in a Tiny Subspace","created":"2018-12-12","cited_by_count":217},{"id":"2003.02218","title":"The large learning rate phase of deep learning: the catapult mechanism","created":"2020-03-04","cited_by_count":216},{"id":"1901.09401","title":"SGD: General Analysis and Improved Rates","created":"2019-01-27","cited_by_count":207},{"id":"1810.10667","title":"Truncated Back-propagation for Bilevel Optimization","created":"2018-10-25","cited_by_count":190},{"id":"1902.06015","title":"Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit","created":"2019-02-16","cited_by_count":187},{"id":"2301.11235","title":"Handbook of Convergence Theorems for (Stochastic) Gradient Methods","created":"2023-01-26","cited_by_count":163},{"id":"1907.04595","title":"Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks","created":"2019-07-10","cited_by_count":162},{"id":"1611.07476","title":"Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond","created":"2016-11-22","cited_by_count":159},{"id":"2209.15594","title":"Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability","created":"2022-09-30","cited_by_count":155},{"id":"2002.09572","title":"The Break-Even Point on Optimization Trajectories of Deep Neural Networks","created":"2020-02-21","cited_by_count":150},{"id":"2006.14548","title":"Tensor Programs II: Neural Tangent Kernel for Any Architecture","created":"2020-06-25","cited_by_count":143},{"id":"1706.03662","title":"Practical Gauss-Newton Optimisation for Deep Learning","created":"2017-06-12","cited_by_count":142},{"id":"2206.03126","title":"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse","created":"2022-06-07","cited_by_count":141},{"id":"1808.05587","title":"Deep Convolutional Networks as shallow Gaussian Processes","created":"2018-08-16","cited_by_count":138},{"id":"2205.09745","title":"Understanding Gradient Descent on Edge of Stability in Deep Learning","created":"2022-05-19","cited_by_count":128},{"id":"1909.05989","title":"Finite Depth and Width Corrections to the Neural Tangent Kernel","created":"2019-09-13","cited_by_count":115},{"id":"2205.09653","title":"Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks","created":"2022-05-19","cited_by_count":105},{"id":"2207.14484","title":"Adaptive Gradient Methods at the Edge of Stability","created":"2022-07-29","cited_by_count":101},{"id":"2310.02244","title":"Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks","created":"2023-10-03","cited_by_count":97},{"id":"1807.05031","title":"On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length","created":"2018-07-13","cited_by_count":88},{"id":"2204.01050","title":"Understanding the unstable convergence of gradient descent","created":"2022-04-03","cited_by_count":86},{"id":"1910.00019","title":"Non-Gaussian processes and neural networks at finite widths","created":"2019-09-30","cited_by_count":78},{"id":"2309.16620","title":"Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit","created":"2023-09-28","cited_by_count":66},{"id":"2210.03294","title":"Understanding Edge-of-Stability Training Dynamics with a Minimalist Example","created":"2022-10-07","cited_by_count":56},{"id":"2106.16225","title":"Analytic Insights into Structure and Rank of Neural Network Hessian Maps","created":"2021-06-30","cited_by_count":52},{"id":"1812.05994","title":"Products of Many Large Random Matrices and Gradients in Deep Neural Networks","created":"2018-12-14","cited_by_count":49},{"id":"2010.12859","title":"Stable ResNet","created":"2020-10-24","cited_by_count":49},{"id":"2306.17759","title":"The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit","created":"2023-06-30","cited_by_count":47},{"id":"2304.03408","title":"Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks","created":"2023-04-06","cited_by_count":43},{"id":"2206.02768","title":"The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization","created":"2022-06-06","cited_by_count":41},{"id":"2305.18411","title":"Feature-Learning Networks Are Consistent Across Widths At Realistic Scales","created":"2023-05-28","cited_by_count":41},{"id":"2210.04860","title":"Second-order regression models exhibit progressive sharpening to the edge of stability","created":"2022-10-10","cited_by_count":40},{"id":"2308.01814","title":"Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit","created":"2023-08-03","cited_by_count":38},{"id":"1502.02127","title":"Hyperparameter Search in Machine Learning","created":"2015-02-07","cited_by_count":33},{"id":"2106.04013","title":"The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization","created":"2021-06-07","cited_by_count":33},{"id":"2106.06615","title":"Precise characterization of the prior predictive distribution of deep ReLU networks","created":"2021-06-11","cited_by_count":28},{"id":"2307.04204","title":"Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory","created":"2023-07-09","cited_by_count":26},{"id":"2210.00688","title":"On the infinite-depth limit of finite-width neural networks","created":"2022-10-03","cited_by_count":22},{"id":"2311.02076","title":"Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos","created":"2023-11-03","cited_by_count":22},{"id":"2302.00453","title":"Width and Depth Limits Commute in Residual Networks","created":"2023-02-01","cited_by_count":21},{"id":"2305.04684","title":"ASDL: A Unified Interface for Gradient Preconditioning in PyTorch","created":"2023-05-08","cited_by_count":21},{"id":"2210.04909","title":"Meta-Principled Family of Hyperparameter Scaling Strategies","created":"2022-10-10","cited_by_count":19},{"id":"2302.12250","title":"Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width","created":"2023-02-23","cited_by_count":19},{"id":"2102.12810","title":"Hyperparameter Transfer Learning with Adaptive Complexity","created":"2021-02-25","cited_by_count":17},{"id":"2010.13117","title":"Hyperparameter Transfer Across Developer Adjustments","created":"2020-10-25","cited_by_count":11},{"id":"2212.07295","title":"Maximal Initial Learning Rates in Deep ReLU Networks","created":"2022-12-14","cited_by_count":11},{"id":"2305.07810","title":"Depth Dependence of $\\mu$P Learning Rates in ReLU MLPs","created":"2023-05-13","cited_by_count":9},{"id":"2310.12079","title":"Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks","created":"2023-10-18","cited_by_count":4},{"id":"2312.00209","title":"On the Interplay Between Stepsize Tuning and Progressive Sharpening","created":"2023-11-30","cited_by_count":4},{"id":"2106.03763","title":"Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks","created":"2021-06-07","cited_by_count":3}],"next_cursor":null}