{"total":679,"rows":[{"id":"2202.10054","title":"Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution","created":"2022-02-21","cited_by_count":739,"area":null},{"id":"2106.07139","title":"Pre-Trained Models: Past, Present and Future","created":"2021-06-14","cited_by_count":319,"area":null},{"id":"2205.13671","title":"Transformer for Partial Differential Equations' Operator Learning","created":"2022-05-26","cited_by_count":283,"area":null},{"id":"2102.06810","title":"Understanding self-supervised Learning Dynamics without Contrastive Pairs","created":"2021-02-12","cited_by_count":256,"area":null},{"id":"2308.06767","title":"A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations","created":"2023-08-13","cited_by_count":202,"area":null},{"id":"2502.07529","title":"Training Deep Learning Models with Norm-Constrained LMOs","created":"2025-02-11","cited_by_count":192,"area":null},{"id":"2106.06042","title":"FedBABU: Towards Enhanced Representation for Federated Image Classification","created":"2021-06-04","cited_by_count":182,"area":null},{"id":"2306.00802","title":"Birth of a Transformer: A Memory Viewpoint","created":"2023-06-01","cited_by_count":161,"area":null},{"id":"2206.03126","title":"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse","created":"2022-06-07","cited_by_count":141,"area":null},{"id":"2405.00781","title":"Barren Plateaus in Variational Quantum Computing","created":"2024-05-01","cited_by_count":133,"area":null},{"id":"2110.05038","title":"Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs","created":"2021-10-11","cited_by_count":122,"area":null},{"id":"2203.10833","title":"Hyperbolic Vision Transformers: Combining Improvements in Metric Learning","created":"2022-03-21","cited_by_count":111,"area":null},{"id":"2102.09430","title":"State Entropy Maximization with Random Encoders for Efficient Exploration","created":"2021-02-18","cited_by_count":108,"area":null},{"id":"2104.07167","title":"Orthogonalizing Convolutional Layers with the Cayley Transform","created":"2021-04-14","cited_by_count":105,"area":null},{"id":"2205.09653","title":"Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks","created":"2022-05-19","cited_by_count":105,"area":null},{"id":"2206.04041","title":"Neural Collapse: A Review on Modelling Principles and Generalization","created":"2022-06-08","cited_by_count":95,"area":null},{"id":"2210.06681","title":"Brain Network Transformer","created":"2022-10-13","cited_by_count":94,"area":null},{"id":"2407.16677","title":"From Imitation to Refinement -- Residual RL for Precise Assembly","created":"2024-07-23","cited_by_count":90,"area":null},{"id":"2209.12892","title":"Learning to Learn with Generative Models of Neural Network Checkpoints","created":"2022-09-26","cited_by_count":86,"area":null},{"id":"2106.15933","title":"Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity","created":"2021-06-30","cited_by_count":85,"area":null},{"id":"2103.05633","title":"Proof-of-Learning: Definitions and Practice","created":"2021-03-09","cited_by_count":84,"area":null},{"id":"2105.14602","title":"On the geometry of generalization and memorization in deep neural networks","created":"2021-05-30","cited_by_count":73,"area":null},{"id":"2106.03443","title":"Causal Influence Detection for Improving Efficiency in Reinforcement Learning","created":"2021-06-07","cited_by_count":71,"area":null},{"id":"2307.02129","title":"How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model","created":"2023-07-05","cited_by_count":71,"area":null},{"id":"2201.07395","title":"Overview frequency principle/spectral bias in deep learning","created":"2022-01-19","cited_by_count":69,"area":null},{"id":"2108.10573","title":"The staircase property: How hierarchical structure can guide deep learning","created":"2021-08-24","cited_by_count":62,"area":null},{"id":"2303.00055","title":"Learning time-scales in two-layers neural networks","created":"2023-02-28","cited_by_count":57,"area":null},{"id":"2409.17858","title":"How Feature Learning Can Improve Neural Scaling Laws","created":"2024-09-26","cited_by_count":56,"area":null},{"id":"2105.04550","title":"Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth","created":"2021-05-10","cited_by_count":55,"area":null},{"id":"2106.16225","title":"Analytic Insights into Structure and Rank of Neural Network Hessian Maps","created":"2021-06-30","cited_by_count":52,"area":null},{"id":"2111.05193","title":"A Survey on Green Deep Learning","created":"2021-11-08","cited_by_count":52,"area":null},{"id":"2306.07042","title":"Transformers learn through gradual rank increase","created":"2023-06-12","cited_by_count":52,"area":null},{"id":"2308.12108","title":"The Local Learning Coefficient: A Singularity-Aware Complexity Measure","created":"2023-08-23","cited_by_count":51,"area":null},{"id":"2211.11567","title":"Neural networks trained with SGD learn distributions of increasing complexity","created":"2022-11-21","cited_by_count":50,"area":null},{"id":"2311.00636","title":"Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures","created":"2023-11-01","cited_by_count":49,"area":null},{"id":"2409.14623","title":"From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks","created":"2024-09-22","cited_by_count":49,"area":null},{"id":"2202.07508","title":"Deep Constrained Least Squares for Blind Image Super-Resolution","created":"2022-02-15","cited_by_count":47,"area":null},{"id":"2103.17268","title":"Fast Certified Robust Training with Short Warmup","created":"2021-03-31","cited_by_count":46,"area":null},{"id":"2201.09871","title":"On Evaluation Metrics for Graph Generative Models","created":"2022-01-24","cited_by_count":46,"area":null},{"id":"2501.16265","title":"Training Dynamics of In-Context Learning in Linear Attention","created":"2025-01-27","cited_by_count":46,"area":null},{"id":"2307.00144","title":"Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows","created":"2023-06-30","cited_by_count":45,"area":null},{"id":"2102.08098","title":"GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training","created":"2021-02-16","cited_by_count":44,"area":null},{"id":"2310.09528","title":"Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks","created":"2023-10-14","cited_by_count":44,"area":null},{"id":"2502.10818","title":"On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning","created":"2025-02-15","cited_by_count":43,"area":null},{"id":"2203.06020","title":"Enhancing Adversarial Training with Second-Order Statistics of Weights","created":"2022-03-11","cited_by_count":42,"area":null},{"id":"2406.06158","title":"Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning","created":"2024-06-10","cited_by_count":42,"area":null},{"id":"2103.06460","title":"Recent Advances on Neural Network Pruning at Initialization","created":"2021-03-11","cited_by_count":41,"area":null},{"id":"2407.00143","title":"InfoNCE: Identifying the Gap Between Theory and Practice","created":"2024-06-28","cited_by_count":40,"area":null},{"id":"2201.08025","title":"Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape","created":"2022-01-20","cited_by_count":39,"area":null},{"id":"2208.03160","title":"Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks","created":"2022-08-05","cited_by_count":39,"area":null},{"id":"2306.04251","title":"Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks","created":"2023-06-07","cited_by_count":39,"area":null},{"id":"2311.18823","title":"Initializing Models with Larger Ones","created":"2023-11-30","cited_by_count":39,"area":null},{"id":"2406.01775","title":"OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models","created":"2024-06-03","cited_by_count":39,"area":null},{"id":"2103.08115","title":"Universal Representation Learning of Knowledge Bases by Jointly Embedding Instances and Ontological Concepts","created":"2021-03-15","cited_by_count":37,"area":null},{"id":"2207.10430","title":"The Neural Race Reduction: Dynamics of Abstraction in Gated Networks","created":"2022-07-21","cited_by_count":37,"area":null},{"id":"2306.13575","title":"Scaling MLPs: A Tale of Inductive Bias","created":"2023-06-23","cited_by_count":37,"area":null},{"id":"2311.01906","title":"Simplifying Transformer Blocks","created":"2023-11-03","cited_by_count":37,"area":null},{"id":"2502.19249","title":"Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases","created":"2025-02-26","cited_by_count":37,"area":null},{"id":"2503.21676","title":"How do language models learn facts? Dynamics, curricula and hallucinations","created":"2025-03-27","cited_by_count":37,"area":null},{"id":"2207.02542","title":"Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems","created":"2022-07-06","cited_by_count":36,"area":null},{"id":"2303.15438","title":"On the Stepwise Nature of Self-Supervised Learning","created":"2023-03-27","cited_by_count":36,"area":null},{"id":"2102.04396","title":"SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality","created":"2021-02-08","cited_by_count":35,"area":null},{"id":"2102.07346","title":"On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers","created":"2021-02-15","cited_by_count":35,"area":null},{"id":"2201.12680","title":"Understanding Deep Contrastive Learning via Coordinate-wise Optimization","created":"2022-01-29","cited_by_count":35,"area":null},{"id":"2210.17216","title":"Symmetries, flat minima, and the conserved quantities of gradient flow","created":"2022-10-31","cited_by_count":35,"area":null},{"id":"2301.01452","title":"The Predictive Forward-Forward Algorithm","created":"2023-01-04","cited_by_count":35,"area":null},{"id":"2311.11108","title":"Auxiliary Losses for Learning Generalizable Concept-based Models","created":"2023-11-18","cited_by_count":35,"area":null},{"id":"2107.06608","title":"Continuous vs. Discrete Optimization of Deep Neural Networks","created":"2021-07-14","cited_by_count":34,"area":null},{"id":"2301.02819","title":"ExcelFormer: A neural network surpassing GBDTs on tabular data","created":"2023-01-07","cited_by_count":34,"area":null},{"id":"2406.06811","title":"Learning Continually by Spectral Regularization","created":"2024-06-10","cited_by_count":34,"area":null},{"id":"2106.04110","title":"A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs","created":"2021-06-08","cited_by_count":33,"area":null},{"id":"2203.02925","title":"Weakly Supervised Temporal Action Localization via Representative Snippet Knowledge Propagation","created":"2022-03-06","cited_by_count":32,"area":null},{"id":"2103.07939","title":"Semi-Supervised Video Deraining with Dynamical Rain Generator","created":"2021-03-14","cited_by_count":31,"area":null},{"id":"2408.11804","title":"Approaching Deep Learning through the Spectral Dynamics of Weights","created":"2024-08-21","cited_by_count":31,"area":null},{"id":"2108.05574","title":"Implicit Sparse Regularization: The Impact of Depth and Early Stopping","created":"2021-08-12","cited_by_count":30,"area":null},{"id":"2308.12553","title":"Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy","created":"2023-08-24","cited_by_count":30,"area":null},{"id":"2106.03970","title":"Batch Normalization Orthogonalizes Representations in Deep Random Networks","created":"2021-06-07","cited_by_count":29,"area":null},{"id":"2202.00565","title":"Data-driven emergence of convolutional structure in neural networks","created":"2022-02-01","cited_by_count":29,"area":null},{"id":"2303.04143","title":"Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?","created":"2023-03-07","cited_by_count":29,"area":null},{"id":"2604.21691","title":"There Will Be a Scientific Theory of Deep Learning","created":"2026-04-23","cited_by_count":29,"area":null},{"id":"2110.12661","title":"ZerO Initialization: Initializing Neural Networks with only Zeros and Ones","created":"2021-10-25","cited_by_count":28,"area":null},{"id":"2203.08120","title":"Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers","created":"2022-03-15","cited_by_count":28,"area":null},{"id":"2404.07470","title":"Scalable Language Model with Generalized Continual Learning","created":"2024-04-11","cited_by_count":28,"area":null},{"id":"2108.01538","title":"Geometry of Linear Convolutional Networks","created":"2021-08-03","cited_by_count":27,"area":null},{"id":"2201.11729","title":"Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks","created":"2022-01-27","cited_by_count":27,"area":null},{"id":"2310.20703","title":"Vanishing Gradients in Reinforcement Finetuning of Language Models","created":"2023-10-31","cited_by_count":27,"area":null},{"id":"2410.23819","title":"Weight decay induces low-rank attention layers","created":"2024-10-31","cited_by_count":27,"area":null},{"id":"2101.09048","title":"Selfish Sparse RNN Training","created":"2021-01-22","cited_by_count":26,"area":null},{"id":"2304.03589","title":"On Efficient Training of Large-Scale Deep Learning Models: A Literature Review","created":"2023-04-07","cited_by_count":26,"area":null},{"id":"2306.10426","title":"Understanding Certified Training with Interval Bound Propagation","created":"2023-06-17","cited_by_count":26,"area":null},{"id":"2312.00246","title":"Directions of Curvature as an Explanation for Loss of Plasticity","created":"2023-11-30","cited_by_count":26,"area":null},{"id":"2406.04112","title":"Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation","created":"2024-06-06","cited_by_count":26,"area":null},{"id":"2105.14301","title":"A Theory of Neural Tangent Kernel Alignment and Its Influence on Training","created":"2021-05-29","cited_by_count":25,"area":null},{"id":"2205.10637","title":"Symmetry Teleportation for Accelerated Optimization","created":"2022-05-21","cited_by_count":25,"area":null},{"id":"2209.01634","title":"Single-source Domain Expansion Network for Cross-Scene Hyperspectral Image Classification","created":"2022-09-04","cited_by_count":25,"area":null},{"id":"2303.08046","title":"Ultra-High-Resolution Detector Simulation with Intra-Event Aware GAN and Self-Supervised Relational Reasoning","created":"2023-03-07","cited_by_count":25,"area":null},{"id":"2309.16932","title":"Symmetry Induces Structure and Constraint of Learning","created":"2023-09-29","cited_by_count":25,"area":null},{"id":"2505.17863","title":"The emergence of sparse attention: impact of data distribution and benefits of repetition","created":"2025-05-23","cited_by_count":25,"area":null},{"id":"2107.13289","title":"The loss landscape of deep linear neural networks: a second-order analysis","created":"2021-07-28","cited_by_count":24,"area":null},{"id":"2112.03215","title":"Multi-scale Feature Learning Dynamics: Insights for Double Descent","created":"2021-12-06","cited_by_count":24,"area":null}],"next_cursor":"100"}