{"list":"other","total":68,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1512.03385","title":"Deep Residual Learning for Image Recognition","created":"2015-12-10","cited_by_count":52096},{"id":"2006.11239","title":"Denoising Diffusion Probabilistic Models","created":"2020-06-19","cited_by_count":21076},{"id":"2303.08774","title":"GPT-4 Technical Report","created":"2023-03-15","cited_by_count":18396},{"id":"2203.02155","title":"Training language models to follow instructions with human feedback","created":"2022-03-04","cited_by_count":15971},{"id":"1707.06347","title":"Proximal Policy Optimization Algorithms","created":"2017-07-20","cited_by_count":15215},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2106.09685","title":"LoRA: Low-Rank Adaptation of Large Language Models","created":"2021-06-17","cited_by_count":14380},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"2103.14030","title":"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows","created":"2021-03-25","cited_by_count":12139},{"id":"1609.02907","title":"Semi-Supervised Classification with Graph Convolutional Networks","created":"2016-09-09","cited_by_count":11361},{"id":"1710.10903","title":"Graph Attention Networks","created":"2017-10-30","cited_by_count":8923},{"id":"2306.05685","title":"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena","created":"2023-06-09","cited_by_count":8626},{"id":"2212.09748","title":"Scalable Diffusion Models with Transformers","created":"2022-12-19","cited_by_count":6054},{"id":"2204.02311","title":"PaLM: Scaling Language Modeling with Pathways","created":"2022-04-05","cited_by_count":5326},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"2404.14219","title":"Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone","created":"2024-04-22","cited_by_count":1917},{"id":"1910.02054","title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","created":"2019-10-04","cited_by_count":1301},{"id":"2106.04560","title":"Scaling Vision Transformers","created":"2021-06-08","cited_by_count":999},{"id":"1804.04235","title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost","created":"2018-04-11","cited_by_count":905},{"id":"1604.06174","title":"Training Deep Nets with Sublinear Memory Cost","created":"2016-04-21","cited_by_count":848},{"id":"1503.05671","title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","created":"2015-03-19","cited_by_count":796},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"2403.03507","title":"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection","created":"2024-03-06","cited_by_count":463},{"id":"2302.06675","title":"Symbolic Discovery of Optimization Algorithms","created":"2023-02-13","cited_by_count":434},{"id":"2110.02861","title":"8-bit Optimizers via Block-wise Quantization","created":"2021-10-06","cited_by_count":403},{"id":"1611.01838","title":"Entropy-SGD: Biasing Gradient Descent Into Wide Valleys","created":"2016-11-06","cited_by_count":359},{"id":"2305.17333","title":"Fine-Tuning Language Models with Just Forward Passes","created":"2023-05-27","cited_by_count":345},{"id":"1905.11881","title":"Why gradient clipping accelerates training: A theoretical justification for adaptivity","created":"2019-05-28","cited_by_count":341},{"id":"1901.10159","title":"An Investigation into Neural Net Optimization via Hessian Eigenvalue Density","created":"2019-01-29","cited_by_count":292},{"id":"1912.07145","title":"PyHessian: Neural Networks Through the Lens of the Hessian","created":"2019-12-16","cited_by_count":292},{"id":"1706.04454","title":"Empirical Analysis of the Hessian of Over-Parametrized Neural Networks","created":"2017-06-14","cited_by_count":290},{"id":"1608.05859","title":"Using the Output Embedding to Improve Language Models","created":"2016-08-20","cited_by_count":272},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265},{"id":"2104.07857","title":"ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning","created":"2021-04-16","cited_by_count":256},{"id":"1812.04754","title":"Gradient Descent Happens in a Tiny Subspace","created":"2018-12-12","cited_by_count":217},{"id":"2208.06677","title":"Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models","created":"2022-08-13","cited_by_count":174},{"id":"2405.18392","title":"Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations","created":"2024-05-28","cited_by_count":170},{"id":"2405.15682","title":"The Road Less Scheduled","created":"2024-05-24","cited_by_count":163},{"id":"1611.07476","title":"Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond","created":"2016-11-22","cited_by_count":159},{"id":"2306.09782","title":"Full Parameter Fine-tuning for Large Language Models with Limited Resources","created":"2023-06-16","cited_by_count":134},{"id":"2403.08763","title":"Simple and Scalable Strategies to Continually Pre-train Large Language Models","created":"2024-03-13","cited_by_count":127},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"1907.04164","title":"Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model","created":"2019-07-09","cited_by_count":117},{"id":"2208.09632","title":"Adam Can Converge Without Any Modification On Update Rules","created":"2022-08-20","cited_by_count":114},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"2008.11865","title":"Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra","created":"2020-08-27","cited_by_count":94},{"id":"1802.08241","title":"Hessian-based Analysis of Large Batch Training and Robustness to Adversaries","created":"2018-02-22","cited_by_count":87},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"1901.08244","title":"Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians","created":"2019-01-24","cited_by_count":68},{"id":"2309.01507","title":"Memory Efficient Optimizers with 4-bit States","created":"2023-09-04","cited_by_count":54},{"id":"2404.08080","title":"Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models","created":"2024-04-11","cited_by_count":54},{"id":"2306.00204","title":"Toward Understanding Why Adam Converges Faster Than SGD for Transformers","created":"2023-05-31","cited_by_count":53},{"id":"2010.04261","title":"Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks","created":"2020-10-08","cited_by_count":45},{"id":"2012.03801","title":"A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization","created":"2020-12-07","cited_by_count":44},{"id":"2307.02047","title":"CAME: Confidence-guided Adaptive Memory Efficient Optimization","created":"2023-07-05","cited_by_count":36},{"id":"2103.01519","title":"Hessian Eigenspectra of More Realistic Nonlinear Models","created":"2021-03-02","cited_by_count":33},{"id":"2310.10195","title":"AdaLomo: Low-memory Optimization with Adaptive Learning Rate","created":"2023-10-16","cited_by_count":31},{"id":"1810.13108","title":"A general system of differential equations to model first order adaptive algorithms","created":"2018-10-31","cited_by_count":28},{"id":"2402.01567","title":"Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise","created":"2024-02-02","cited_by_count":26},{"id":"2211.02254","title":"How Does Adaptive Optimization Impact Local Neural Network Geometry?","created":"2022-11-04","cited_by_count":24},{"id":"2102.07227","title":"Learning by Turning: Neural Architecture Aware Optimisation","created":"2021-02-14","cited_by_count":23},{"id":"2402.07114","title":"Towards Quantifying the Preconditioning Effect of Adam","created":"2024-02-11","cited_by_count":19},{"id":"2401.10809","title":"Neglected Hessian component explains mysteries in Sharpness regularization","created":"2024-01-19","cited_by_count":12},{"id":"1905.11286","title":"Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks","created":"2019-05-27","cited_by_count":9},{"id":"2209.00809","title":"Optimal Diagonal Preconditioning","created":"2022-09-02","cited_by_count":7},{"id":"1905.09899","title":"Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning","created":"2019-05-23","cited_by_count":6}],"next_cursor":null}