{"list":"other","total":85,"rows":[{"id":"1412.6980","title":"Adam: A Method for Stochastic Optimization","created":"2014-12-22","cited_by_count":37240},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"2104.09864","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","created":"2021-04-20","cited_by_count":4703},{"id":"1606.08415","title":"Gaussian Error Linear Units (GELUs)","created":"2016-06-27","cited_by_count":3502},{"id":"1806.07572","title":"Neural Tangent Kernel: Convergence and Generalization in Neural Networks","created":"2018-06-20","cited_by_count":2621},{"id":"2309.17453","title":"Efficient Streaming Language Models with Attention Sinks","created":"2023-09-29","cited_by_count":2186},{"id":"2108.12409","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","created":"2021-08-27","cited_by_count":1103},{"id":"2504.13837","title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","created":"2025-04-18","cited_by_count":982},{"id":"1702.03118","title":"Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning","created":"2017-02-10","cited_by_count":950},{"id":"1804.04235","title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost","created":"2018-04-11","cited_by_count":905},{"id":"1904.09237","title":"On the Convergence of Adam and Beyond","created":"2019-04-19","cited_by_count":841},{"id":"1902.06720","title":"Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent","created":"2019-02-18","cited_by_count":738},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"1812.07956","title":"On Lazy Training in Differentiable Programming","created":"2018-12-19","cited_by_count":675},{"id":"1810.02054","title":"Gradient Descent Provably Optimizes Over-parameterized Neural Networks","created":"2018-10-04","cited_by_count":659},{"id":"1904.11955","title":"On Exact Computation with an Infinitely Wide Neural Net","created":"2019-04-26","cited_by_count":573},{"id":"1901.08584","title":"Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks","created":"2019-01-24","cited_by_count":566},{"id":"1804.06561","title":"A Mean Field View of the Landscape of Two-Layers Neural Networks","created":"2018-04-18","cited_by_count":466},{"id":"2403.03507","title":"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection","created":"2024-03-06","cited_by_count":463},{"id":"2302.06675","title":"Symbolic Discovery of Optimization Algorithms","created":"2023-02-13","cited_by_count":434},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"2305.10429","title":"DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining","created":"2023-05-17","cited_by_count":399},{"id":"1811.04918","title":"Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers","created":"2018-11-12","cited_by_count":340},{"id":"2305.19466","title":"The Impact of Positional Encoding on Length Generalization in Transformers","created":"2023-05-31","cited_by_count":327},{"id":"1704.04289","title":"Stochastic Gradient Descent as Approximate Bayesian Inference","created":"2017-04-13","cited_by_count":300},{"id":"1703.00887","title":"How to Escape Saddle Points Efficiently","created":"2017-03-02","cited_by_count":285},{"id":"2010.07468","title":"AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients","created":"2020-10-15","cited_by_count":283},{"id":"2102.01293","title":"Scaling Laws for Transfer","created":"2021-02-02","cited_by_count":272},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265},{"id":"1909.12673","title":"A Constructive Prediction of the Generalization Error Across Scales","created":"2019-09-27","cited_by_count":210},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192},{"id":"2502.07529","title":"Training Deep Learning Models with Norm-Constrained LMOs","created":"2025-02-11","cited_by_count":192},{"id":"1902.06015","title":"Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit","created":"2019-02-16","cited_by_count":187},{"id":"2312.03863","title":"Efficient Large Language Models: A Survey","created":"2023-12-06","cited_by_count":186},{"id":"2403.16952","title":"Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance","created":"2024-03-25","cited_by_count":184},{"id":"2411.12372","title":"RedPajama: an Open Dataset for Training Large Language Models","created":"2024-11-19","cited_by_count":182},{"id":"2407.01492","title":"RegMix: Data Mixture as Regression for Language Model Pre-training","created":"2024-07-01","cited_by_count":177},{"id":"1912.02803","title":"Neural Tangents: Fast and Easy Infinite Neural Networks in Python","created":"2019-12-05","cited_by_count":149},{"id":"2406.16793","title":"Adam-mini: Use Fewer Learning Rates To Gain More","created":"2024-06-24","cited_by_count":137},{"id":"2010.01092","title":"On the linearity of large non-linear models: when and why the tangent kernel is constant","created":"2020-10-02","cited_by_count":129},{"id":"2407.13623","title":"Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies","created":"2024-07-18","cited_by_count":118},{"id":"1808.09372","title":"Mean Field Analysis of Neural Networks: A Central Limit Theorem","created":"2018-08-28","cited_by_count":107},{"id":"1803.01206","title":"On the Power of Over-parametrization in Neural Networks with Quadratic Activation","created":"2018-03-03","cited_by_count":102},{"id":"2509.02046","title":"Fantastic Pretraining Optimizers and Where to Find Them","created":"2025-09-02","cited_by_count":100},{"id":"1607.07819","title":"Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with $ \\ell^1 $ and $ \\ell^0 $ Controls","created":"2016-07-26","cited_by_count":84},{"id":"2510.05491","title":"NorMuon: Making Muon more efficient and scalable","created":"2025-10-07","cited_by_count":84},{"id":"2410.05192","title":"Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective","created":"2024-10-07","cited_by_count":81},{"id":"2504.05295","title":"Dion: Distributed Orthonormalized Updates","created":"2025-04-07","cited_by_count":68},{"id":"2509.01440","title":"Benchmarking Optimizers for Large Language Model Pretraining","created":"2025-09-01","cited_by_count":68},{"id":"2412.05270","title":"APOLLO: SGD-like Memory, AdamW-level Performance","created":"2024-12-06","cited_by_count":54},{"id":"2507.09404","title":"Scaling Laws for Optimal Data Mixtures","created":"2025-07-12","cited_by_count":52},{"id":"1810.06801","title":"Quasi-hyperbolic momentum and Adam for deep learning","created":"2018-10-16","cited_by_count":48},{"id":"2402.01258","title":"Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape","created":"2024-02-02","cited_by_count":46},{"id":"1903.04440","title":"Mean Field Analysis of Deep Neural Networks","created":"2019-03-11","cited_by_count":42},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41},{"id":"2510.16981","title":"MuonBP: Faster Muon via Block-Periodic Orthogonalization","created":"2025-10-19","cited_by_count":35},{"id":"2508.10047","title":"A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions","created":"2025-08-12","cited_by_count":31},{"id":"2512.01868","title":"The Mean-Field Dynamics of Transformers","created":"2025-12-01","cited_by_count":30},{"id":"2407.17465","title":"u-$\\mu$P: The Unit-Scaled Maximal Update Parametrization","created":"2024-07-24","cited_by_count":29},{"id":"2510.01631","title":"Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls","created":"2025-10-02","cited_by_count":26},{"id":"2208.13614","title":"Neural Tangent Kernel: A Survey","created":"2022-08-29","cited_by_count":23},{"id":"2602.05400","title":"OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration","created":"2026-02-05","cited_by_count":21},{"id":"2507.12466","title":"Language Models Improve When Pretraining Data Matches Target Tasks","created":"2025-07-16","cited_by_count":19},{"id":"2510.09378","title":"The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton","created":"2025-10-10","cited_by_count":17},{"id":"2604.09967","title":"Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning","created":"2026-04-11","cited_by_count":17},{"id":"2502.12120","title":"LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws","created":"2025-02-17","cited_by_count":16},{"id":"2402.11537","title":"Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning","created":"2024-02-18","cited_by_count":15},{"id":"2603.20527","title":"RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization","created":"2026-03-20","cited_by_count":15},{"id":"2505.24844","title":"Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning","created":"2025-05-30","cited_by_count":14},{"id":"2602.17080","title":"Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum","created":"2026-02-19","cited_by_count":14},{"id":"2403.14917","title":"Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective","created":"2024-03-22","cited_by_count":13},{"id":"2503.03862","title":"Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions","created":"2025-03-05","cited_by_count":12},{"id":"2603.10067","title":"HTMuon: Improving Muon via Heavy-Tailed Spectral Correction","created":"2026-03-10","cited_by_count":12},{"id":"2401.02643","title":"Training and Serving System of Foundation Models: A Comprehensive Survey","created":"2024-01-05","cited_by_count":11},{"id":"2111.00856","title":"Large-Scale Deep Learning Optimizations: A Comprehensive Survey","created":"2021-11-01","cited_by_count":10},{"id":"2404.05728","title":"An Empirical Study of $\\mu$P Learning Rate Transfer","created":"2024-04-08","cited_by_count":10},{"id":"2510.25817","title":"A Survey on Efficient Large Language Model Training: From Data-centric Perspectives","created":"2025-10-29","cited_by_count":9},{"id":"2602.07712","title":"Towards Robust Scaling Laws for Optimizers","created":"2026-02-07","cited_by_count":6},{"id":"2602.13498","title":"TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers","created":"2026-02-13","cited_by_count":6},{"id":"2510.03313","title":"Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining","created":"2025-09-30","cited_by_count":5},{"id":"2406.07247","title":"Dynamical Mean-Field Theory of Self-Attention Neural Networks","created":"2024-06-11","cited_by_count":4},{"id":"2510.18245","title":"Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs","created":"2025-10-21","cited_by_count":4},{"id":"2603.17970","title":"Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training","created":"2026-03-18","cited_by_count":4},{"id":"2603.00416","title":"MuonRec: Shifting the Optimizer Paradigm Beyond Adam in Scalable Generative Recommendation","created":"2026-02-28","cited_by_count":2}],"next_cursor":null}