{"total":109,"rows":[{"id":"2403.03507","title":"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection","created":"2024-03-06","cited_by_count":463,"area":null},{"id":"2409.20325","title":"Old Optimizer, New Norm: An Anthology","created":"2024-09-30","cited_by_count":193,"area":null},{"id":"2502.07529","title":"Training Deep Learning Models with Norm-Constrained LMOs","created":"2025-02-11","cited_by_count":192,"area":null},{"id":"2505.16932","title":"The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm","created":"2025-05-22","cited_by_count":110,"area":null},{"id":"2505.02222","title":"Practical Efficiency of Muon for Pretraining","created":"2025-05-04","cited_by_count":80,"area":null},{"id":"2410.21265","title":"Modular Duality in Deep Learning","created":"2024-10-28","cited_by_count":77,"area":null},{"id":"2405.14813","title":"Scalable Optimization in the Modular Norm","created":"2024-05-23","cited_by_count":68,"area":null},{"id":"2504.05295","title":"Dion: Distributed Orthonormalized Updates","created":"2025-04-07","cited_by_count":68,"area":null},{"id":"2502.08606","title":"Distillation Scaling Laws","created":"2025-02-12","cited_by_count":67,"area":"lr"},{"id":"2503.20762","title":"ASGO: Adaptive Structured Gradient Optimization","created":"2025-03-26","cited_by_count":57,"area":null},{"id":"2312.16903","title":"Spike No More: Stabilizing the Pre-training of Large Language Models","created":"2023-12-28","cited_by_count":52,"area":null},{"id":"2509.26030","title":"Muon Outperforms Adam in Tail-End Associative Memory Learning","created":"2025-09-30","cited_by_count":46,"area":null},{"id":"2406.06158","title":"Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning","created":"2024-06-10","cited_by_count":42,"area":null},{"id":"2405.15712","title":"Infinite Limits of Multi-head Transformer Dynamics","created":"2024-05-24","cited_by_count":39,"area":null},{"id":"2601.08393","title":"Controlled LLM Training on Spectral Sphere","created":"2026-01-13","cited_by_count":31,"area":"moe"},{"id":"2407.17465","title":"u-$\\mu$P: The Unit-Scaled Maximal Update Parametrization","created":"2024-07-24","cited_by_count":29,"area":null},{"id":"2406.06248","title":"Compute Better Spent: Replacing Dense Layers with Structured Matrices","created":"2024-06-10","cited_by_count":28,"area":null},{"id":"2405.21064","title":"Recurrent neural networks: vanishing and exploding gradients are not the end of the story","created":"2024-05-31","cited_by_count":25,"area":null},{"id":"2410.23922","title":"Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training","created":"2024-10-31","cited_by_count":22,"area":null},{"id":"2604.03444","title":"Olmo Hybrid: From Theory to Practice and Back","created":"2026-04-03","cited_by_count":22,"area":null},{"id":"2512.05620","title":"Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales","created":"2025-12-05","cited_by_count":21,"area":null},{"id":"2507.13338","title":"Training Transformers with Enforced Lipschitz Constants","created":"2025-07-17","cited_by_count":20,"area":null},{"id":"2510.19093","title":"Weight Decay may matter more than muP for Learning Rate Transfer in Practice","created":"2025-10-21","cited_by_count":20,"area":null},{"id":"2505.24749","title":"SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training","created":"2025-05-30","cited_by_count":19,"area":null},{"id":"2506.07254","title":"A Stable Whitening Optimizer for Efficient Neural Network Training","created":"2025-06-08","cited_by_count":18,"area":null},{"id":"2410.17881","title":"AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning","created":"2024-10-23","cited_by_count":17,"area":null},{"id":"2603.09697","title":"Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning","created":"2026-03-10","cited_by_count":17,"area":null},{"id":"2405.15743","title":"Sparse maximal update parameterization: A holistic approach to sparse training dynamics","created":"2024-05-24","cited_by_count":16,"area":null},{"id":"2606.16899","title":"Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization","created":"2026-06-15","cited_by_count":16,"area":null},{"id":"2410.20634","title":"Plastic Learning with Deep Fourier Features","created":"2024-10-27","cited_by_count":14,"area":null},{"id":"2410.02117","title":"Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices","created":"2024-10-03","cited_by_count":13,"area":"moe"},{"id":"2410.04642","title":"The Optimization Landscape of SGD Across the Feature Learning Strength","created":"2024-10-06","cited_by_count":13,"area":null},{"id":"2601.20205","title":"Hyperparameter Transfer with Mixture-of-Expert Layers","created":"2026-01-28","cited_by_count":13,"area":"moe"},{"id":"2603.04964","title":"Replaying pre-training data improves fine-tuning","created":"2026-03-05","cited_by_count":13,"area":null},{"id":"2506.01260","title":"Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism","created":"2025-06-02","cited_by_count":12,"area":null},{"id":"2601.23000","title":"Mano: Restriking Manifold Optimization for LLM Training","created":"2026-01-30","cited_by_count":12,"area":null},{"id":"2603.09952","title":"On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer","created":"2026-03-10","cited_by_count":12,"area":null},{"id":"2311.18718","title":"The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks","created":"2023-11-30","cited_by_count":11,"area":null},{"id":"2502.03210","title":"From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning","created":"2025-02-05","cited_by_count":10,"area":null},{"id":"2410.05838","title":"Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit","created":"2024-10-08","cited_by_count":9,"area":null},{"id":"2605.08980","title":"Muon Does Not Converge on Convex Lipschitz Functions","created":"2026-05-09","cited_by_count":9,"area":null},{"id":"2510.03871","title":"Optimal Scaling Needs Optimal Norm","created":"2025-10-04","cited_by_count":8,"area":null},{"id":"2510.15262","title":"Robust Layerwise Scaling Rules by Proper Weight Decay Tuning","created":"2025-10-17","cited_by_count":8,"area":null},{"id":"2602.09314","title":"Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory","created":"2026-02-10","cited_by_count":8,"area":null},{"id":"2608.22764","title":"LpWM: A Case for Sparse Representations in World Models","created":"2026-08-24","cited_by_count":8,"area":"wm"},{"id":"2509.23500","title":"Beyond Outliers: A Study of Optimizers Under Quantization","created":"2025-09-27","cited_by_count":6,"area":null},{"id":"2602.12429","title":"Stabilizing Native Low-Rank LLM Pretraining","created":"2026-02-12","cited_by_count":6,"area":null},{"id":"2605.09238","title":"Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds","created":"2026-05-10","cited_by_count":6,"area":null},{"id":"2605.10797","title":"Muown: Row-Norm Control for Muon Optimization","created":"2026-05-11","cited_by_count":6,"area":null},{"id":"2605.12492","title":"Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation","created":"2026-05-12","cited_by_count":6,"area":null},{"id":"2502.05967","title":"$\\mu$nit Scaling: Simple and Scalable FP8 LLM Training","created":"2025-02-09","cited_by_count":5,"area":null},{"id":"2502.19571","title":"LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM","created":"2025-02-26","cited_by_count":5,"area":null},{"id":"2509.24218","title":"Conda: Column-Normalized Adam for Training Large Language Models Faster","created":"2025-09-29","cited_by_count":5,"area":null},{"id":"2605.06654","title":"Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less","created":"2026-05-07","cited_by_count":5,"area":null},{"id":"2605.15871","title":"Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design","created":"2026-05-15","cited_by_count":5,"area":"agents"},{"id":"2502.17405","title":"Function-Space Learning Rates","created":"2025-02-24","cited_by_count":4,"area":null},{"id":"2503.09565","title":"Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\\mu$P Parametrization","created":"2025-03-12","cited_by_count":4,"area":null},{"id":"2507.08091","title":"Low-rank Momentum Factorization for Memory Efficient Training","created":"2025-07-10","cited_by_count":4,"area":null},{"id":"2510.06627","title":"POME: Post Optimization Model Edit via Muon-style Projection","created":"2025-10-08","cited_by_count":4,"area":null},{"id":"2604.09258","title":"Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima","created":"2026-04-10","cited_by_count":4,"area":null},{"id":"2501.08727","title":"Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models","created":"2025-01-15","cited_by_count":3,"area":null},{"id":"2508.09752","title":"$\\mu$-Parametrization for Mixture of Experts","created":"2025-08-13","cited_by_count":3,"area":"moe"},{"id":"2509.17866","title":"Understanding Post-Training Structural Changes in Large Language Models","created":"2025-09-22","cited_by_count":3,"area":null},{"id":"2510.09103","title":"AdaPM: a Partial Momentum Algorithm for LLM Training","created":"2025-10-10","cited_by_count":3,"area":null},{"id":"2511.21377","title":"Controlling changes to attention logits","created":"2025-11-26","cited_by_count":3,"area":null},{"id":"2604.27077","title":"Learning Rate Transfer in Normalized Transformers","created":"2026-04-29","cited_by_count":3,"area":null},{"id":"2606.09658","title":"Muon Learns More Robust and Transferable Features than Adam","created":"2026-06-08","cited_by_count":3,"area":null},{"id":"2606.23637","title":"Muown Implicitly Performs Angular Step-size Decay","created":"2026-06-22","cited_by_count":3,"area":null},{"id":"2606.30509","title":"Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics","created":"2026-06-29","cited_by_count":3,"area":null},{"id":"2404.19719","title":"The lazy (NTK) and rich ($\\mu$P) regimes: a gentle tutorial","created":"2024-04-30","cited_by_count":2,"area":null},{"id":"2507.09768","title":"Knowing When to Quit: Probabilistic Early Exits for Speech Separation","created":"2025-07-13","cited_by_count":2,"area":"inference"},{"id":"2510.04202","title":"Spectral Alignment as Predictor of Loss Explosion in Neural Network Training","created":"2025-10-05","cited_by_count":2,"area":null},{"id":"2601.04890","title":"Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers","created":"2026-01-08","cited_by_count":2,"area":null},{"id":"2605.18528","title":"Tight Complexity of Muon under Heavy-Tailed Noise","created":"2026-05-18","cited_by_count":2,"area":null},{"id":"2605.21803","title":"Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws","created":"2026-05-20","cited_by_count":2,"area":null},{"id":"2605.26459","title":"MuCon: Clipped Muon Updates for LLM Training","created":"2026-05-26","cited_by_count":2,"area":null},{"id":"2607.10517","title":"Conditional Optimal Bridge for Riemannian Activation Steering","created":"2026-07-12","cited_by_count":2,"area":"mechinterp"},{"id":"2607.17620","title":"PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer","created":"2026-07-20","cited_by_count":2,"area":null},{"id":"2607.23777","title":"Scale Weight Decay and Train Better","created":"2026-07-26","cited_by_count":2,"area":null},{"id":"2511.04981","title":"Scaling depth capacity via zero/one-layer model expansion","created":"2025-11-07","cited_by_count":1,"area":null},{"id":"2512.06010","title":"Fast and Flexible Robustness Certificates for Semantic Segmentation","created":"2025-12-03","cited_by_count":1,"area":null},{"id":"2604.02505","title":"Optimal Projection-Free Adaptive SGD for Matrix Optimization","created":"2026-04-02","cited_by_count":1,"area":null},{"id":"2605.04418","title":"Demystifying Manifold Constraints in LLM Pre-training","created":"2026-05-06","cited_by_count":1,"area":null},{"id":"2605.15290","title":"GQA-{\\mu}P: The maximal parameterization update for grouped query attention","created":"2026-05-14","cited_by_count":1,"area":null},{"id":"2605.29387","title":"On the Optimizer Dependence of Neural Scaling Laws","created":"2026-05-28","cited_by_count":1,"area":null},{"id":"2606.05484","title":"Learned Subspace Compression for Communication-Efficient Pipeline Parallelism","created":"2026-06-03","cited_by_count":1,"area":null},{"id":"2606.06418","title":"Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss","created":"2026-06-04","cited_by_count":1,"area":null},{"id":"2606.06470","title":"PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training","created":"2026-06-04","cited_by_count":1,"area":null},{"id":"2608.25990","title":"Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon","created":"2026-08-26","cited_by_count":1,"area":null},{"id":"2509.07963","title":"Customizing the Inductive Biases of Softmax Attention using Structured Matrices","created":"2025-09-09","cited_by_count":0,"area":null},{"id":"2601.01306","title":"Towards a Principled Muon under $\\mu\\mathsf{P}$: Ensuring Spectral Conditions throughout Training","created":"2026-01-04","cited_by_count":0,"area":null},{"id":"2602.01845","title":"No Generation without Representation: Efficient Causal Protein Language Models Enable Zero-Shot Fitness Estimation","created":"2026-02-02","cited_by_count":0,"area":null},{"id":"2602.20937","title":"Extending $\\mu$P: Spectral Conditions for Feature Learning Across Optimizers","created":"2026-02-24","cited_by_count":0,"area":null},{"id":"2605.10164","title":"Hyperparameter Transfer for Dense Associative Memories","created":"2026-05-11","cited_by_count":0,"area":null},{"id":"2605.14200","title":"How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization","created":"2026-05-13","cited_by_count":0,"area":"moe"},{"id":"2605.20314","title":"Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases","created":"2026-05-19","cited_by_count":0,"area":null},{"id":"2605.28975","title":"A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio","created":"2026-05-27","cited_by_count":0,"area":null},{"id":"2606.04048","title":"Unlocking Feature Learning in Gated Delta Networks at Scale","created":"2026-06-02","cited_by_count":0,"area":null},{"id":"2607.06592","title":"LipSSD: Lipschitz-Constrained Single-Shot Detection for Adversarially Robust Object Detection","created":"2026-07-06","cited_by_count":0,"area":null},{"id":"2607.25200","title":"Algorithmic Separation between Constant-Depth and Logarithmic-Depth Neural Networks","created":"2026-07-28","cited_by_count":0,"area":null}],"next_cursor":"100"}