{"total":206,"rows":[{"id":"2101.12176","title":"On the Origin of Implicit Regularization in Stochastic Gradient Descent","created":"2021-01-28","cited_by_count":184,"area":null},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104,"area":null},{"id":"2106.14343","title":"High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails","created":"2021-06-28","cited_by_count":80,"area":null},{"id":"2102.12470","title":"On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)","created":"2021-02-24","cited_by_count":73,"area":null},{"id":"2111.13171","title":"Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks","created":"2021-11-25","cited_by_count":70,"area":null},{"id":"2207.02628","title":"The alignment property of SGD noise and how it helps select flat minima: A stability analysis","created":"2022-07-06","cited_by_count":64,"area":null},{"id":"2105.13462","title":"On Linear Stability of SGD and Input-Smoothness of Neural Networks","created":"2021-05-27","cited_by_count":63,"area":null},{"id":"2109.14119","title":"Stochastic Training is Not Necessary for Generalization","created":"2021-09-29","cited_by_count":63,"area":null},{"id":"2410.13849","title":"From Gradient Clipping to Normalization for Heavy Tailed SGD","created":"2024-10-17","cited_by_count":49,"area":null},{"id":"2106.00042","title":"A study on the plasticity of neural networks","created":"2021-05-31","cited_by_count":44,"area":null},{"id":"2106.03795","title":"Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks","created":"2021-06-07","cited_by_count":44,"area":null},{"id":"2202.02831","title":"Anticorrelated Noise Injection for Improved Generalization","created":"2022-02-06","cited_by_count":42,"area":null},{"id":"2412.19529","title":"Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping","created":"2024-12-27","cited_by_count":42,"area":null},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40,"area":null},{"id":"2102.05375","title":"Strength of Minibatch Noise in SGD","created":"2021-02-10","cited_by_count":39,"area":null},{"id":"2201.08025","title":"Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape","created":"2022-01-20","cited_by_count":39,"area":null},{"id":"2103.07601","title":"Approximating How Single Head Attention Learns","created":"2021-03-13","cited_by_count":38,"area":null},{"id":"2303.02749","title":"Revisiting the Noise Model of Stochastic Gradient Descent","created":"2023-03-05","cited_by_count":36,"area":null},{"id":"2102.04396","title":"SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality","created":"2021-02-08","cited_by_count":35,"area":null},{"id":"2102.10346","title":"Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance","created":"2021-02-20","cited_by_count":35,"area":null},{"id":"2107.11228","title":"Taxonomizing local versus global structure in neural network loss landscapes","created":"2021-07-23","cited_by_count":34,"area":null},{"id":"2102.10264","title":"On Proximal Policy Optimization's Heavy-tailed Gradients","created":"2021-02-20","cited_by_count":31,"area":null},{"id":"2104.10719","title":"A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection","created":"2021-04-21","cited_by_count":31,"area":null},{"id":"2105.09557","title":"Power-law escape rate of SGD","created":"2021-05-20","cited_by_count":29,"area":null},{"id":"2102.09385","title":"Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes","created":"2021-02-16","cited_by_count":28,"area":null},{"id":"2103.17182","title":"Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization","created":"2021-03-31","cited_by_count":28,"area":null},{"id":"2501.06842","title":"SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training","created":"2025-01-12","cited_by_count":28,"area":null},{"id":"2108.00781","title":"Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers","created":"2021-08-02","cited_by_count":27,"area":null},{"id":"2302.06763","title":"Breaking the Lower Bound with (Little) Structure: Acceleration in Non-Convex Stochastic Optimization with Heavy-Tailed Noise","created":"2023-02-14","cited_by_count":27,"area":null},{"id":"2106.02588","title":"Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis","created":"2021-06-04","cited_by_count":25,"area":null},{"id":"2202.11632","title":"Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variance","created":"2022-02-23","cited_by_count":24,"area":null},{"id":"2103.04902","title":"Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem","created":"2021-03-08","cited_by_count":23,"area":null},{"id":"2106.04881","title":"Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms","created":"2021-06-09","cited_by_count":23,"area":null},{"id":"2301.08203","title":"An SDE for Modeling SAM: Theory and Insights","created":"2023-01-19","cited_by_count":22,"area":null},{"id":"2305.03938","title":"Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees","created":"2023-05-06","cited_by_count":22,"area":null},{"id":"2405.14578","title":"Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling","created":"2024-05-23","cited_by_count":22,"area":null},{"id":"2303.12277","title":"Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptation","created":"2023-03-22","cited_by_count":20,"area":null},{"id":"2312.00359","title":"Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training","created":"2023-12-01","cited_by_count":20,"area":null},{"id":"2107.09133","title":"The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion","created":"2021-07-19","cited_by_count":19,"area":null},{"id":"2208.02304","title":"How Much Privacy Does Federated Learning with Secure Aggregation Guarantee?","created":"2022-08-03","cited_by_count":19,"area":null},{"id":"2410.16561","title":"Revisiting Gradient Normalization and Clipping for Nonconvex SGD under Heavy-Tailed Noise: Necessity, Sufficiency, and Acceleration","created":"2024-10-21","cited_by_count":18,"area":null},{"id":"2508.04860","title":"Can SGD Handle Heavy-Tailed Noise?","created":"2025-08-06","cited_by_count":17,"area":null},{"id":"2508.07473","title":"Online Convex Optimization with Heavy Tails: Old Algorithms, New Regrets, and Applications","created":"2025-08-10","cited_by_count":17,"area":null},{"id":"2201.13011","title":"On the Power-Law Hessian Spectrums in Deep Learning","created":"2022-01-31","cited_by_count":16,"area":null},{"id":"2305.05611","title":"Metric Space Magnitude and Generalisation in Neural Networks","created":"2023-05-09","cited_by_count":15,"area":null},{"id":"2102.04297","title":"Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise","created":"2021-02-08","cited_by_count":14,"area":null},{"id":"2212.02083","title":"On the Overlooked Structure of Stochastic Gradients","created":"2022-12-05","cited_by_count":13,"area":null},{"id":"2502.07923","title":"Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under $(L_0, L_1)$-Smoothness","created":"2025-02-11","cited_by_count":13,"area":null},{"id":"2102.07006","title":"Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections","created":"2021-02-13","cited_by_count":12,"area":null},{"id":"2106.08414","title":"On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control","created":"2021-06-15","cited_by_count":12,"area":null},{"id":"2201.06714","title":"AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization","created":"2022-01-18","cited_by_count":11,"area":null},{"id":"2211.10691","title":"Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States","created":"2022-11-19","cited_by_count":11,"area":null},{"id":"2405.13453","title":"A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacy","created":"2024-05-22","cited_by_count":11,"area":null},{"id":"2506.11214","title":"Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise","created":"2025-06-12","cited_by_count":11,"area":null},{"id":"2210.00690","title":"Taming Fat-Tailed (\"Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learning","created":"2022-10-03","cited_by_count":10,"area":null},{"id":"2506.03470","title":"Models of Heavy-Tailed Mechanistic Universality","created":"2025-06-04","cited_by_count":10,"area":null},{"id":"2506.06280","title":"Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias","created":"2025-06-06","cited_by_count":10,"area":null},{"id":"2603.14315","title":"Enhancing LLM Training via Spectral Clipping","created":"2026-03-15","cited_by_count":10,"area":null},{"id":"2603.15958","title":"Deriving Hyperparameter Scaling Laws via Modern Optimization Theory","created":"2026-03-16","cited_by_count":10,"area":null},{"id":"2205.11361","title":"Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent","created":"2022-05-23","cited_by_count":9,"area":null},{"id":"2310.00692","title":"A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent","created":"2023-10-01","cited_by_count":9,"area":null},{"id":"2404.17442","title":"Uniform Generalization Bounds on Data-Dependent Hypothesis Sets via PAC-Bayesian Theory on Random Sets","created":"2024-04-26","cited_by_count":9,"area":null},{"id":"2406.04657","title":"From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks","created":"2024-06-07","cited_by_count":9,"area":null},{"id":"2105.14677","title":"Characterization of Generalizability of Spike Timing Dependent Plasticity trained Spiking Neural Networks","created":"2021-05-31","cited_by_count":8,"area":null},{"id":"2302.11618","title":"Heterogeneous Neuronal and Synaptic Dynamics for Spike-Efficient Unsupervised Learning: Theory and Design Principles","created":"2023-02-22","cited_by_count":8,"area":null},{"id":"2605.06884","title":"Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition","created":"2026-05-07","cited_by_count":8,"area":null},{"id":"2309.03004","title":"A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness","created":"2023-09-06","cited_by_count":7,"area":null},{"id":"2310.18784","title":"High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise","created":"2023-10-28","cited_by_count":7,"area":null},{"id":"2404.05555","title":"On the Convergence of Continual Learning with Adaptive Methods","created":"2024-04-08","cited_by_count":7,"area":null},{"id":"2410.15637","title":"Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetry","created":"2024-10-21","cited_by_count":7,"area":null},{"id":"2411.00999","title":"Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers","created":"2024-11-01","cited_by_count":7,"area":null},{"id":"2502.04164","title":"Efficient Distributed Optimization under Heavy-Tailed Noise","created":"2025-02-06","cited_by_count":7,"area":null},{"id":"2602.06500","title":"Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?","created":"2026-02-06","cited_by_count":7,"area":null},{"id":"2106.11514","title":"Rethinking Adam: A Twofold Exponential Moving Average Approach","created":"2021-06-22","cited_by_count":6,"area":null},{"id":"2201.02313","title":"Stochastic Saddle Point Problems with Decision-Dependent Distributions","created":"2022-01-07","cited_by_count":6,"area":null},{"id":"2205.06689","title":"Heavy-Tail Phenomenon in Decentralized SGD","created":"2022-05-13","cited_by_count":6,"area":null},{"id":"2207.04922","title":"On uniform-in-time diffusion approximation for stochastic gradient descent","created":"2022-07-11","cited_by_count":6,"area":null},{"id":"2303.10434","title":"Byzantine-Resilient Federated Learning at Edge","created":"2023-03-18","cited_by_count":6,"area":null},{"id":"2111.04004","title":"Exponential escape efficiency of SGD from sharp minima in non-stationary regime","created":"2021-11-07","cited_by_count":5,"area":null},{"id":"2201.04545","title":"On generalization bounds for deep networks based on loss surface implicit regularization","created":"2022-01-12","cited_by_count":5,"area":null},{"id":"2206.08558","title":"How You Start Matters for Generalization","created":"2022-06-17","cited_by_count":5,"area":null},{"id":"2209.09162","title":"On the Theoretical Properties of Noise Correlation in Stochastic Optimization","created":"2022-09-19","cited_by_count":5,"area":null},{"id":"2304.12579","title":"Learning Trajectories are Generalization Indicators","created":"2023-04-25","cited_by_count":5,"area":null},{"id":"2306.08125","title":"Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD","created":"2023-06-13","cited_by_count":5,"area":null},{"id":"2306.10299","title":"Edge Intelligence Over the Air: Two Faces of Interference in Federated Learning","created":"2023-06-17","cited_by_count":5,"area":null},{"id":"2307.13680","title":"High Probability Analysis for Non-Convex Stochastic Optimization with Clipping","created":"2023-07-25","cited_by_count":5,"area":null},{"id":"2405.17529","title":"Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails","created":"2024-05-27","cited_by_count":5,"area":null},{"id":"2509.12527","title":"Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Design","created":"2025-09-16","cited_by_count":5,"area":null},{"id":"2109.09833","title":"Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics","created":"2021-09-20","cited_by_count":4,"area":null},{"id":"2112.00980","title":"Trap of Feature Diversity in the Learning of MLPs","created":"2021-12-02","cited_by_count":4,"area":null},{"id":"2312.00427","title":"From Mutual Information to Expected Dynamics: New Generalization Bounds for Heavy-Tailed SGD","created":"2023-12-01","cited_by_count":4,"area":null},{"id":"2402.01382","title":"Emergence of heavy tails in homogenized stochastic gradient descent","created":"2024-02-02","cited_by_count":4,"area":null},{"id":"2410.16340","title":"Limit Theorems for Stochastic Gradient Descent with Infinite Variance","created":"2024-10-21","cited_by_count":4,"area":null},{"id":"2506.00181","title":"On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach","created":"2025-05-30","cited_by_count":4,"area":null},{"id":"2509.14952","title":"Stochastic Bilevel Optimization with Heavy-Tailed Noise","created":"2025-09-18","cited_by_count":4,"area":null},{"id":"2510.11676","title":"Accelerated stochastic first-order method for convex optimization under heavy-tailed noise","created":"2025-10-13","cited_by_count":4,"area":null},{"id":"2203.14434","title":"Flexible risk design using bi-directional dispersion","created":"2022-03-28","cited_by_count":3,"area":null},{"id":"2309.05337","title":"Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems","created":"2023-09-11","cited_by_count":3,"area":null},{"id":"2310.18455","title":"Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent","created":"2023-10-27","cited_by_count":3,"area":null},{"id":"2404.08624","title":"Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks","created":"2024-04-12","cited_by_count":3,"area":null}],"next_cursor":"100"}