{"total":216,"rows":[{"id":"2107.07075","title":"Deep Learning on a Data Diet: Finding Important Examples Early in Training","created":"2021-07-15","cited_by_count":533,"area":null},{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427,"area":null},{"id":"2102.06701","title":"Explaining Neural Scaling Laws","created":"2021-02-12","cited_by_count":363,"area":null},{"id":"2303.14186","title":"TRAK: Attributing Model Behavior at Scale","created":"2023-03-24","cited_by_count":283,"area":null},{"id":"2107.13034","title":"Dataset Distillation with Infinitely Wide Convolutional Networks","created":"2021-07-27","cited_by_count":218,"area":null},{"id":"2103.12828","title":"Learning to Optimize: A Primer and A Benchmark","created":"2021-03-23","cited_by_count":208,"area":null},{"id":"2101.12176","title":"On the Origin of Implicit Regularization in Stochastic Gradient Descent","created":"2021-01-28","cited_by_count":184,"area":null},{"id":"2303.01486","title":"Understanding plasticity in neural networks","created":"2023-03-02","cited_by_count":174,"area":null},{"id":"2209.15594","title":"Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability","created":"2022-09-30","cited_by_count":155,"area":null},{"id":"2205.01445","title":"High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation","created":"2022-05-03","cited_by_count":150,"area":null},{"id":"2205.09745","title":"Understanding Gradient Descent on Edge of Stability in Deep Learning","created":"2022-05-19","cited_by_count":128,"area":null},{"id":"2309.07311","title":"Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs","created":"2023-09-13","cited_by_count":124,"area":null},{"id":"2205.09653","title":"Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks","created":"2022-05-19","cited_by_count":105,"area":null},{"id":"2207.14484","title":"Adaptive Gradient Methods at the Edge of Stability","created":"2022-07-29","cited_by_count":101,"area":null},{"id":"2206.04817","title":"The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon","created":"2022-06-10","cited_by_count":94,"area":null},{"id":"2204.01050","title":"Understanding the unstable convergence of gradient descent","created":"2022-04-03","cited_by_count":86,"area":null},{"id":"2206.07085","title":"Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction","created":"2022-06-14","cited_by_count":86,"area":null},{"id":"2301.02679","title":"Grokking modular arithmetic","created":"2023-01-06","cited_by_count":83,"area":null},{"id":"2210.05337","title":"SGD with Large Step Sizes Learns Sparse Features","created":"2022-10-11","cited_by_count":68,"area":null},{"id":"2305.18270","title":"How Two-Layer Neural Networks Learn, One (Giant) Step at a Time","created":"2023-05-29","cited_by_count":68,"area":null},{"id":"2405.14813","title":"Scalable Optimization in the Modular Norm","created":"2024-05-23","cited_by_count":68,"area":null},{"id":"2201.05125","title":"GradMax: Growing Neural Networks using Gradient Information","created":"2022-01-13","cited_by_count":66,"area":null},{"id":"2109.14119","title":"Stochastic Training is Not Necessary for Generalization","created":"2021-09-29","cited_by_count":63,"area":null},{"id":"2207.12678","title":"Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability","created":"2022-07-26","cited_by_count":63,"area":null},{"id":"2206.10369","title":"The State of Sparse Training in Deep Reinforcement Learning","created":"2022-06-17","cited_by_count":61,"area":null},{"id":"2301.11300","title":"ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients","created":"2023-01-26","cited_by_count":58,"area":null},{"id":"2407.01800","title":"Normalization and effective learning rates in reinforcement learning","created":"2024-07-01","cited_by_count":58,"area":"deeprl"},{"id":"2206.10654","title":"On the Maximum Hessian Eigenvalue and Generalization","created":"2022-06-21","cited_by_count":57,"area":null},{"id":"2303.03108","title":"Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization","created":"2023-03-03","cited_by_count":57,"area":null},{"id":"2210.03294","title":"Understanding Edge-of-Stability Training Dynamics with a Minimalist Example","created":"2022-10-07","cited_by_count":56,"area":null},{"id":"2112.15383","title":"Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs","created":"2021-12-31","cited_by_count":53,"area":null},{"id":"2212.07469","title":"Learning threshold neurons via the \"edge of stability\"","created":"2022-12-14","cited_by_count":51,"area":null},{"id":"2105.02716","title":"Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks","created":"2021-05-06","cited_by_count":50,"area":null},{"id":"2110.03677","title":"Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect","created":"2021-10-07","cited_by_count":50,"area":null},{"id":"2409.14623","title":"From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks","created":"2024-09-22","cited_by_count":49,"area":null},{"id":"2406.09405","title":"Why Warmup the Learning Rate? Underlying Mechanisms and Improvements","created":"2024-06-13","cited_by_count":46,"area":null},{"id":"2304.03408","title":"Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks","created":"2023-04-06","cited_by_count":43,"area":null},{"id":"2305.11788","title":"Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability","created":"2023-05-19","cited_by_count":42,"area":null},{"id":"2406.06158","title":"Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning","created":"2024-06-10","cited_by_count":42,"area":null},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41,"area":null},{"id":"2110.04369","title":"A Loss Curvature Perspective on Training Instability in Deep Learning","created":"2021-10-08","cited_by_count":40,"area":null},{"id":"2210.04860","title":"Second-order regression models exhibit progressive sharpening to the edge of stability","created":"2022-10-10","cited_by_count":40,"area":null},{"id":"2102.05375","title":"Strength of Minibatch Noise in SGD","created":"2021-02-10","cited_by_count":39,"area":null},{"id":"2306.13253","title":"Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok","created":"2023-06-23","cited_by_count":35,"area":null},{"id":"2107.06608","title":"Continuous vs. Discrete Optimization of Deep Neural Networks","created":"2021-07-14","cited_by_count":34,"area":null},{"id":"2305.19753","title":"The Tunnel Effect: Building Data Representations in Deep Neural Networks","created":"2023-05-31","cited_by_count":34,"area":null},{"id":"2110.14402","title":"Learning where to learn: Gradient sparsity in meta and continual learning","created":"2021-10-27","cited_by_count":33,"area":null},{"id":"2303.03428","title":"Towards provably efficient quantum algorithms for large-scale machine-learning models","created":"2023-03-06","cited_by_count":32,"area":null},{"id":"2306.04815","title":"Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning","created":"2023-06-07","cited_by_count":31,"area":null},{"id":"2211.06506","title":"Spectral Evolution and Invariance in Linear-width Neural Networks","created":"2022-11-11","cited_by_count":30,"area":null},{"id":"2202.00980","title":"Robust Training of Neural Networks Using Scale Invariant Architectures","created":"2022-02-02","cited_by_count":29,"area":null},{"id":"2104.11044","title":"Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes","created":"2021-04-22","cited_by_count":28,"area":null},{"id":"2107.05802","title":"How many degrees of freedom do we need to train deep networks: a loss landscape perspective","created":"2021-07-13","cited_by_count":28,"area":null},{"id":"2105.10585","title":"Properties of the After Kernel","created":"2021-05-21","cited_by_count":27,"area":null},{"id":"2205.07069","title":"Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties","created":"2022-05-14","cited_by_count":27,"area":null},{"id":"2307.11108","title":"Flatness-Aware Minimization for Domain Generalization","created":"2023-07-20","cited_by_count":27,"area":null},{"id":"2110.03922","title":"The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks","created":"2021-10-08","cited_by_count":26,"area":null},{"id":"2212.13881","title":"Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features","created":"2022-12-28","cited_by_count":25,"area":null},{"id":"2310.17087","title":"Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult","created":"2023-10-26","cited_by_count":25,"area":null},{"id":"2211.02254","title":"How Does Adaptive Optimization Impact Local Neural Network Geometry?","created":"2022-11-04","cited_by_count":24,"area":null},{"id":"2302.08692","title":"SAM operates far from home: eigenvalue regularization as a dynamical phenomenon","created":"2023-02-17","cited_by_count":24,"area":null},{"id":"2206.07252","title":"Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions","created":"2022-06-15","cited_by_count":23,"area":null},{"id":"2402.17457","title":"Super Consistency of Neural Network Landscapes and Learning Rate Transfer","created":"2024-02-27","cited_by_count":23,"area":null},{"id":"2409.15156","title":"Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling","created":"2024-09-23","cited_by_count":23,"area":"lr"},{"id":"2311.02076","title":"Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos","created":"2023-11-03","cited_by_count":22,"area":null},{"id":"2412.01505","title":"Scaling Law for Language Models Training Considering Batch Size","created":"2024-12-02","cited_by_count":22,"area":null},{"id":"2403.01874","title":"A Survey on Evaluation of Out-of-Distribution Generalization","created":"2024-03-04","cited_by_count":21,"area":null},{"id":"2312.00359","title":"Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training","created":"2023-12-01","cited_by_count":20,"area":null},{"id":"2209.03695","title":"Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes","created":"2022-09-08","cited_by_count":19,"area":null},{"id":"2302.12250","title":"Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width","created":"2023-02-23","cited_by_count":19,"area":null},{"id":"2503.18595","title":"Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition","created":"2025-03-24","cited_by_count":19,"area":null},{"id":"2405.17580","title":"Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes","created":"2024-05-27","cited_by_count":18,"area":null},{"id":"2205.11787","title":"Quadratic models for understanding catapult dynamics of neural networks","created":"2022-05-24","cited_by_count":17,"area":null},{"id":"2205.15809","title":"Feature Learning in $L_{2}$-regularized DNNs: Attraction/Repulsion and Sparsity","created":"2022-05-31","cited_by_count":17,"area":null},{"id":"2310.01687","title":"From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression","created":"2023-10-02","cited_by_count":17,"area":null},{"id":"2405.13456","title":"Deep linear networks for regression are implicitly regularized towards flat minima","created":"2024-05-22","cited_by_count":17,"area":null},{"id":"2305.12133","title":"Loss Spike in Training Neural Networks","created":"2023-05-20","cited_by_count":16,"area":null},{"id":"2307.06887","title":"Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks","created":"2023-07-13","cited_by_count":16,"area":null},{"id":"2410.04887","title":"Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse","created":"2024-10-07","cited_by_count":16,"area":null},{"id":"2510.06213","title":"Training Dynamics Impact Post-Training Quantization Robustness","created":"2025-10-07","cited_by_count":15,"area":null},{"id":"2210.12030","title":"Evolution of Neural Tangent Kernels under Benign and Adversarial Training","created":"2022-10-21","cited_by_count":14,"area":null},{"id":"2406.05033","title":"Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes","created":"2024-06-07","cited_by_count":14,"area":null},{"id":"2506.06489","title":"Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks","created":"2025-06-06","cited_by_count":14,"area":null},{"id":"2607.21716","title":"A Defense of the Quadratic Model","created":"2026-07-23","cited_by_count":14,"area":null},{"id":"2311.04163","title":"Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization","created":"2023-11-07","cited_by_count":13,"area":null},{"id":"2402.05173","title":"Towards Understanding Inductive Bias in Transformers: A View From Infinity","created":"2024-02-07","cited_by_count":13,"area":null},{"id":"2403.18494","title":"Learning in PINNs: Phase transition, diffusion equilibrium, and generalization","created":"2024-03-27","cited_by_count":13,"area":null},{"id":"2410.04642","title":"The Optimization Landscape of SGD Across the Feature Learning Strength","created":"2024-10-06","cited_by_count":13,"area":null},{"id":"2506.04805","title":"Adaptive Preconditioners Trigger Loss Spikes in Adam","created":"2025-06-05","cited_by_count":13,"area":null},{"id":"2109.07573","title":"Differentiable Physics: A Position Piece","created":"2021-09-14","cited_by_count":12,"area":null},{"id":"2309.10688","title":"On the different regimes of Stochastic Gradient Descent","created":"2023-09-19","cited_by_count":12,"area":null},{"id":"2406.03260","title":"Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers","created":"2024-06-05","cited_by_count":12,"area":null},{"id":"2406.06838","title":"Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes","created":"2024-06-10","cited_by_count":12,"area":null},{"id":"2410.04489","title":"Grokking at the Edge of Linear Separability","created":"2024-10-06","cited_by_count":12,"area":null},{"id":"2103.12682","title":"How to decay your learning rate","created":"2021-03-23","cited_by_count":11,"area":null},{"id":"2212.07295","title":"Maximal Initial Learning Rates in Deep ReLU Networks","created":"2022-12-14","cited_by_count":11,"area":null},{"id":"2302.01952","title":"On a continuous time model of gradient descent dynamics and instability in deep learning","created":"2023-02-03","cited_by_count":11,"area":null},{"id":"2201.11137","title":"Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization","created":"2022-01-26","cited_by_count":10,"area":null},{"id":"2204.09664","title":"Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?","created":"2022-04-20","cited_by_count":10,"area":null},{"id":"2205.15142","title":"Special Properties of Gradient Descent with Large Learning Rates","created":"2022-05-30","cited_by_count":10,"area":null}],"next_cursor":"100"}