{"list":"other","total":72,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"2302.13971","title":"LLaMA: Open and Efficient Foundation Language Models","created":"2023-02-27","cited_by_count":14132},{"id":"1502.03167","title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","created":"2015-02-11","cited_by_count":7005},{"id":"1607.06450","title":"Layer Normalization","created":"2016-07-21","cited_by_count":5669},{"id":"2312.00752","title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","created":"2023-12-01","cited_by_count":4718},{"id":"2104.09864","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","created":"2021-04-20","cited_by_count":4703},{"id":"2205.14135","title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","created":"2022-05-27","cited_by_count":3673},{"id":"2205.01068","title":"OPT: Open Pre-trained Transformer Language Models","created":"2022-05-02","cited_by_count":3512},{"id":"1806.07572","title":"Neural Tangent Kernel: Convergence and Generalization in Neural Networks","created":"2018-06-20","cited_by_count":2621},{"id":"2304.01373","title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","created":"2023-04-03","cited_by_count":1786},{"id":"1904.10509","title":"Generating Long Sequences with Sparse Transformers","created":"2019-04-23","cited_by_count":1546},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"2211.10438","title":"SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models","created":"2022-11-18","cited_by_count":1386},{"id":"2112.11446","title":"Scaling Language Models: Methods, Analysis & Insights from Training Gopher","created":"2021-12-08","cited_by_count":1202},{"id":"2002.04745","title":"On Layer Normalization in the Transformer Architecture","created":"2020-02-12","cited_by_count":922},{"id":"2210.02414","title":"GLM-130B: An Open Bilingual Pre-trained Model","created":"2022-10-05","cited_by_count":843},{"id":"2103.17239","title":"Going deeper with Image Transformers","created":"2021-03-31","cited_by_count":764},{"id":"2302.05442","title":"Scaling Vision Transformers to 22 Billion Parameters","created":"2023-02-10","cited_by_count":694},{"id":"1812.07956","title":"On Lazy Training in Differentiable Programming","created":"2018-12-19","cited_by_count":675},{"id":"2404.00456","title":"QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs","created":"2024-03-30","cited_by_count":634},{"id":"1711.00165","title":"Deep Neural Networks as Gaussian Processes","created":"2017-11-01","cited_by_count":622},{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427},{"id":"2103.03404","title":"Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth","created":"2021-03-05","cited_by_count":419},{"id":"2102.06171","title":"High-Performance Large-Scale Image Recognition Without Normalization","created":"2021-02-11","cited_by_count":360},{"id":"1804.11271","title":"Gaussian Process Behaviour in Wide Deep Neural Networks","created":"2018-04-30","cited_by_count":317},{"id":"2012.09816","title":"Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning","created":"2020-12-17","cited_by_count":297},{"id":"2402.17762","title":"Massive Activations in Large Language Models","created":"2024-02-27","cited_by_count":294},{"id":"2010.04245","title":"Query-Key Normalization for Transformers","created":"2020-10-08","cited_by_count":280},{"id":"2003.04887","title":"ReZero is All You Need: Fast Convergence at Large Depth","created":"2020-03-10","cited_by_count":264},{"id":"1606.05340","title":"Exponential expressivity in deep neural networks through transient chaos","created":"2016-06-16","cited_by_count":261},{"id":"1907.12009","title":"Representation Degeneration Problem in Training Natural Language Generation Models","created":"2019-07-28","cited_by_count":255},{"id":"1809.10853","title":"Adaptive Input Representations for Neural Language Modeling","created":"2018-09-28","cited_by_count":226},{"id":"1901.09321","title":"Fixup Initialization: Residual Learning Without Normalization","created":"2019-01-27","cited_by_count":198},{"id":"1611.01232","title":"Deep Information Propagation","created":"2016-11-04","cited_by_count":195},{"id":"2011.14522","title":"Feature Learning in Infinite-Width Neural Networks","created":"2020-11-30","cited_by_count":190},{"id":"2209.13325","title":"Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models","created":"2022-09-27","cited_by_count":181},{"id":"1806.05393","title":"Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks","created":"2018-06-14","cited_by_count":176},{"id":"2303.06296","title":"Stabilizing Transformer Training by Preventing Attention Entropy Collapse","created":"2023-03-11","cited_by_count":161},{"id":"2109.04404","title":"All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality","created":"2021-09-09","cited_by_count":156},{"id":"1910.12478","title":"Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes","created":"2019-10-28","cited_by_count":151},{"id":"2306.12929","title":"Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing","created":"2023-06-22","cited_by_count":151},{"id":"2205.01445","title":"High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation","created":"2022-05-03","cited_by_count":150},{"id":"2109.12948","title":"Understanding and Overcoming the Challenges of Efficient Transformer Quantization","created":"2021-09-27","cited_by_count":141},{"id":"2206.03126","title":"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse","created":"2022-06-07","cited_by_count":141},{"id":"1711.04735","title":"Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice","created":"2017-11-13","cited_by_count":132},{"id":"2006.10540","title":"Infinite attention: NNGP and NTK for deep attention networks","created":"2020-06-18","cited_by_count":121},{"id":"2105.06990","title":"BERT Busters: Outlier Dimensions that Disrupt Transformers","created":"2021-05-14","cited_by_count":112},{"id":"2205.09653","title":"Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks","created":"2022-05-19","cited_by_count":105},{"id":"1902.06853","title":"On the Impact of the Activation Function on Deep Neural Networks Training","created":"2019-02-19","cited_by_count":102},{"id":"2310.02244","title":"Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks","created":"2023-10-03","cited_by_count":97},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"2304.13013","title":"Stable and low-precision training for large-scale vision-language models","created":"2023-04-25","cited_by_count":75},{"id":"2309.16620","title":"Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit","created":"2023-09-28","cited_by_count":66},{"id":"1902.08129","title":"A Mean Field Theory of Batch Normalization","created":"2019-02-21","cited_by_count":58},{"id":"2205.11380","title":"Outliers Dimensions that Disrupt Transformers Are Driven by Frequency","created":"2022-05-23","cited_by_count":55},{"id":"2110.01765","title":"Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping","created":"2021-10-05","cited_by_count":52},{"id":"2310.00535","title":"JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention","created":"2023-10-01","cited_by_count":52},{"id":"2010.12859","title":"Stable ResNet","created":"2020-10-24","cited_by_count":49},{"id":"2306.17759","title":"The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit","created":"2023-06-30","cited_by_count":47},{"id":"2302.10322","title":"Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation","created":"2023-02-20","cited_by_count":45},{"id":"2206.02768","title":"The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization","created":"2022-06-06","cited_by_count":41},{"id":"2311.01906","title":"Simplifying Transformer Blocks","created":"2023-11-03","cited_by_count":37},{"id":"2106.06615","title":"Precise characterization of the prior predictive distribution of deep ReLU networks","created":"2021-06-11","cited_by_count":28},{"id":"2203.08120","title":"Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers","created":"2022-03-15","cited_by_count":28},{"id":"2404.03605","title":"Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization","created":"2024-04-04","cited_by_count":23},{"id":"2110.11749","title":"Feature Learning and Signal Propagation in Deep Neural Networks","created":"2021-10-22","cited_by_count":18},{"id":"2310.16764","title":"ConvNets Match Vision Transformers at Scale","created":"2023-10-25","cited_by_count":16},{"id":"2305.18399","title":"On the impact of activation and normalization in obtaining isometric embeddings at initialization","created":"2023-05-28","cited_by_count":13},{"id":"2309.11197","title":"The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute","created":"2023-09-20","cited_by_count":10},{"id":"2310.02012","title":"Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion","created":"2023-10-03","cited_by_count":8},{"id":"2310.14036","title":"On discretisation drift and smoothness regularisation in neural network training","created":"2023-10-21","cited_by_count":1}],"next_cursor":null}