{"total":73,"rows":[{"id":"2505.02222","title":"Practical Efficiency of Muon for Pretraining","created":"2025-05-04","cited_by_count":80,"area":null},{"id":"2504.05295","title":"Dion: Distributed Orthonormalized Updates","created":"2025-04-07","cited_by_count":68,"area":null},{"id":"2503.14858","title":"1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities","created":"2025-03-19","cited_by_count":45,"area":"deeprl"},{"id":"2505.13738","title":"Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training","created":"2025-05-19","cited_by_count":43,"area":null},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41,"area":null},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40,"area":null},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35,"area":null},{"id":"2512.10858","title":"Scaling Behavior of Discrete Diffusion Language Models","created":"2025-12-11","cited_by_count":35,"area":"diffusion"},{"id":"2608.30320","title":"On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability","created":"2026-08-31","cited_by_count":35,"area":"moe"},{"id":"2503.04715","title":"Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining","created":"2025-03-06","cited_by_count":34,"area":null},{"id":"2507.01598","title":"Convergence Bound and Critical Batch Size of Muon Optimizer","created":"2025-07-02","cited_by_count":28,"area":null},{"id":"2505.23971","title":"Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training","created":"2025-05-29","cited_by_count":23,"area":null},{"id":"2507.17634","title":"WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training","created":"2025-07-23","cited_by_count":23,"area":null},{"id":"2507.12466","title":"Language Models Improve When Pretraining Data Matches Target Tasks","created":"2025-07-16","cited_by_count":19,"area":null},{"id":"2510.09378","title":"The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton","created":"2025-10-10","cited_by_count":17,"area":null},{"id":"2506.12543","title":"Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling","created":"2025-06-14","cited_by_count":15,"area":null},{"id":"2607.21716","title":"A Defense of the Quadratic Model","created":"2026-07-23","cited_by_count":14,"area":null},{"id":"2509.23678","title":"Towards a Comprehensive Scaling Law of Mixture-of-Experts","created":"2025-09-28","cited_by_count":10,"area":"moe"},{"id":"2602.03702","title":"Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging","created":"2026-02-03","cited_by_count":10,"area":null},{"id":"2603.15958","title":"Deriving Hyperparameter Scaling Laws via Modern Optimization Theory","created":"2026-03-16","cited_by_count":10,"area":null},{"id":"2507.09846","title":"Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training","created":"2025-07-14","cited_by_count":9,"area":null},{"id":"2509.25087","title":"Scaling with Collapse: Efficient and Predictable Training of LLM Families","created":"2025-09-29","cited_by_count":9,"area":null},{"id":"2503.05931","title":"Training and Inference Efficiency of Encoder-Decoder Speech Models","created":"2025-03-07","cited_by_count":8,"area":null},{"id":"2510.03871","title":"Optimal Scaling Needs Optimal Norm","created":"2025-10-04","cited_by_count":8,"area":null},{"id":"2603.21191","title":"On the Role of Batch Size in Stochastic Conditional Gradient Methods","created":"2026-03-22","cited_by_count":7,"area":null},{"id":"2605.19095","title":"ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models","created":"2026-05-18","cited_by_count":7,"area":null},{"id":"2506.15025","title":"Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size","created":"2025-06-17","cited_by_count":6,"area":null},{"id":"2510.14717","title":"Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling","created":"2025-10-16","cited_by_count":6,"area":null},{"id":"2512.08894","title":"Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training","created":"2025-12-09","cited_by_count":6,"area":null},{"id":"2602.14208","title":"Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws","created":"2026-02-15","cited_by_count":6,"area":null},{"id":"2603.12151","title":"IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL","created":"2026-03-12","cited_by_count":6,"area":null},{"id":"2604.12887","title":"VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization","created":"2026-04-14","cited_by_count":6,"area":null},{"id":"2604.21215","title":"The Recurrent Transformer: Greater Effective Depth and Efficient Decoding","created":"2026-04-23","cited_by_count":6,"area":null},{"id":"2604.15297","title":"Benchmarking Optimizers for MLPs in Tabular Deep Learning","created":"2026-04-16","cited_by_count":5,"area":null},{"id":"2412.21124","title":"Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism","created":"2024-12-30","cited_by_count":4,"area":null},{"id":"2505.22549","title":"DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models","created":"2025-05-28","cited_by_count":4,"area":null},{"id":"2509.25049","title":"Efficient Hyperparameter Tuning via Trajectory Invariance Principle","created":"2025-09-29","cited_by_count":4,"area":null},{"id":"2602.01642","title":"The Effect of Mini-Batch Noise on the Implicit Bias of Adam","created":"2026-02-02","cited_by_count":4,"area":null},{"id":"2602.09492","title":"Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA","created":"2026-02-10","cited_by_count":4,"area":null},{"id":"2602.11137","title":"Weight Decay Improves Language Model Plasticity","created":"2026-02-11","cited_by_count":4,"area":null},{"id":"2508.00180","title":"EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes","created":"2025-07-31","cited_by_count":3,"area":null},{"id":"2511.01734","title":"A Proof of Learning Rate Transfer under $\\mu$P","created":"2025-11-03","cited_by_count":3,"area":null},{"id":"2605.23061","title":"Anytime Training with Schedule-Free Spectral Optimization","created":"2026-05-21","cited_by_count":3,"area":null},{"id":"2509.25380","title":"Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs","created":"2025-09-29","cited_by_count":2,"area":null},{"id":"2510.05361","title":"MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates","created":"2025-10-06","cited_by_count":2,"area":null},{"id":"2602.10300","title":"Configuration-to-Performance Scaling Law with Neural Ansatz","created":"2026-02-10","cited_by_count":2,"area":null},{"id":"2605.17787","title":"Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates","created":"2026-05-18","cited_by_count":2,"area":null},{"id":"2606.19179","title":"Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods","created":"2026-06-17","cited_by_count":2,"area":null},{"id":"2606.29158","title":"On the Nonlinearity of Learning Rate Scaling for LLM Training","created":"2026-06-28","cited_by_count":2,"area":null},{"id":"2502.14123","title":"Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression","created":"2025-02-19","cited_by_count":1,"area":"diffusion"},{"id":"2602.04396","title":"LoRDO: Distributed Low-Rank Optimization with Infrequent Communication","created":"2026-02-04","cited_by_count":1,"area":null},{"id":"2602.15327","title":"Prescriptive Scaling Reveals the Evolution of Language Model Capabilities","created":"2026-02-17","cited_by_count":1,"area":null},{"id":"2606.14259","title":"Beyond a Single Explanation of the Adam--SGD Gap","created":"2026-06-12","cited_by_count":1,"area":null},{"id":"2607.01487","title":"How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size","created":"2026-07-01","cited_by_count":1,"area":null},{"id":"2607.27731","title":"Towards joint scaling laws with optimal batch size schedules","created":"2026-07-30","cited_by_count":1,"area":null},{"id":"2602.14462","title":"Silent Inconsistency in Data-Parallel Full Fine-Tuning: Diagnosing Worker-Level Optimization Misalignment","created":"2026-02-16","cited_by_count":0,"area":null},{"id":"2605.09154","title":"Predicting Large Model Test Losses with a Noisy Quadratic System","created":"2026-05-09","cited_by_count":0,"area":null},{"id":"2605.11215","title":"ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload","created":"2026-05-11","cited_by_count":0,"area":null},{"id":"2605.28961","title":"Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions","created":"2026-05-27","cited_by_count":0,"area":null},{"id":"2606.05610","title":"Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training","created":"2026-06-04","cited_by_count":0,"area":null},{"id":"2606.30634","title":"One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining","created":"2026-06-29","cited_by_count":0,"area":null},{"id":"2608.10296","title":"Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension","created":"2026-08-10","cited_by_count":0,"area":null},{"id":"2608.20061","title":"Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts","created":"2026-08-20","cited_by_count":0,"area":"moe"},{"id":"2609.05016","title":"Amortizing Scaling Law Construction Costs","created":"2026-09-04","cited_by_count":0,"area":null},{"id":"2609.08690","title":"Hyperparameter Scaling Laws Across MoE Sparsity","created":"2026-09-08","cited_by_count":0,"area":"moe"},{"id":"2609.19924","title":"The Life of a Token: from Words to Bits on the Wire","created":"2026-09-17","cited_by_count":0,"area":null},{"id":"2609.23055","title":"Optimizers for Diffusion Models: A Controlled Benchmark","created":"2026-09-19","cited_by_count":0,"area":"diffusion"},{"id":"2609.33041","title":"Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons","created":"2026-09-27","cited_by_count":0,"area":null},{"id":"2609.39767","title":"How Does Local Landscape Geometry Evolve in Language Model Pre-Training?","created":"2026-09-30","cited_by_count":0,"area":null},{"id":"2610.02355","title":"Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance","created":"2026-10-01","cited_by_count":0,"area":null},{"id":"2610.06784","title":"How to scale your HEP ML models: A recipe for robust architecture comparisons at scale","created":"2026-10-05","cited_by_count":0,"area":null},{"id":"2610.08975","title":"The Best Optimizer Depends on Batch Size","created":"2026-10-06","cited_by_count":0,"area":null},{"id":"2610.15965","title":"Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs","created":"2026-10-11","cited_by_count":0,"area":null}],"next_cursor":null}