{"list":"other","total":59,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1512.03385","title":"Deep Residual Learning for Image Recognition","created":"2015-12-10","cited_by_count":52096},{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"2302.13971","title":"LLaMA: Open and Efficient Foundation Language Models","created":"2023-02-27","cited_by_count":14132},{"id":"1607.06450","title":"Layer Normalization","created":"2016-07-21","cited_by_count":5669},{"id":"2204.02311","title":"PaLM: Scaling Language Modeling with Pathways","created":"2022-04-05","cited_by_count":5326},{"id":"1804.07461","title":"GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding","created":"2018-04-20","cited_by_count":4899},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"2101.00027","title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","created":"2020-12-31","cited_by_count":2411},{"id":"2006.16236","title":"Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention","created":"2020-06-29","cited_by_count":2022},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"2002.04745","title":"On Layer Normalization in the Transformer Architecture","created":"2020-02-12","cited_by_count":922},{"id":"1503.05671","title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","created":"2015-03-19","cited_by_count":796},{"id":"2103.17239","title":"Going deeper with Image Transformers","created":"2021-03-31","cited_by_count":764},{"id":"1312.6120","title":"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks","created":"2013-12-20","cited_by_count":763},{"id":"1612.08083","title":"Language Modeling with Gated Convolutional Networks","created":"2016-12-23","cited_by_count":702},{"id":"2302.05442","title":"Scaling Vision Transformers to 22 Billion Parameters","created":"2023-02-10","cited_by_count":694},{"id":"1711.00165","title":"Deep Neural Networks as Gaussian Processes","created":"2017-11-01","cited_by_count":622},{"id":"2101.03697","title":"RepVGG: Making VGG-style ConvNets Great Again","created":"2021-01-11","cited_by_count":522},{"id":"2102.11174","title":"Linear Transformers Are Secretly Fast Weight Programmers","created":"2021-02-22","cited_by_count":502},{"id":"2103.03404","title":"Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth","created":"2021-03-05","cited_by_count":419},{"id":"2102.06171","title":"High-Performance Large-Scale Image Recognition Without Normalization","created":"2021-02-11","cited_by_count":360},{"id":"1804.11271","title":"Gaussian Process Behaviour in Wide Deep Neural Networks","created":"2018-04-30","cited_by_count":317},{"id":"2003.04887","title":"ReZero is All You Need: Fast Convergence at Large Depth","created":"2020-03-10","cited_by_count":264},{"id":"1606.05340","title":"Exponential expressivity in deep neural networks through transient chaos","created":"2016-06-16","cited_by_count":261},{"id":"2004.08249","title":"Understanding the Difficulty of Training Transformers","created":"2020-04-17","cited_by_count":210},{"id":"1901.09321","title":"Fixup Initialization: Residual Learning Without Normalization","created":"2019-01-27","cited_by_count":198},{"id":"1611.01232","title":"Deep Information Propagation","created":"2016-11-04","cited_by_count":195},{"id":"2203.00555","title":"DeepNet: Scaling Transformers to 1,000 Layers","created":"2022-03-01","cited_by_count":178},{"id":"1806.05393","title":"Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks","created":"2018-06-14","cited_by_count":176},{"id":"1702.08591","title":"The Shattered Gradients Problem: If resnets are the answer, then what is the question?","created":"2017-02-28","cited_by_count":157},{"id":"1910.12478","title":"Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes","created":"2019-10-28","cited_by_count":151},{"id":"2206.03126","title":"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse","created":"2022-06-07","cited_by_count":141},{"id":"2203.05962","title":"Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice","created":"2022-03-09","cited_by_count":136},{"id":"2206.00133","title":"Pre-training via Denoising for Molecular Property Prediction","created":"2022-05-31","cited_by_count":124},{"id":"1803.01719","title":"How to Start Training: The Effect of Initialization and Architecture","created":"2018-03-05","cited_by_count":109},{"id":"1902.06853","title":"On the Impact of the Activation Function on Deep Neural Networks Training","created":"2019-02-19","cited_by_count":102},{"id":"1907.01470","title":"Augmenting Self-attention with Persistent Memory","created":"2019-07-02","cited_by_count":93},{"id":"2212.14034","title":"Cramming: Training a Language Model on a Single GPU in One Day","created":"2022-12-28","cited_by_count":81},{"id":"2001.05992","title":"Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks","created":"2020-01-16","cited_by_count":73},{"id":"1908.11365","title":"Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention","created":"2019-08-29","cited_by_count":59},{"id":"1902.08129","title":"A Mean Field Theory of Batch Normalization","created":"2019-02-21","cited_by_count":58},{"id":"2305.09828","title":"Mimetic Initialization of Self-Attention Layers","created":"2023-05-16","cited_by_count":54},{"id":"2110.01765","title":"Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping","created":"2021-10-05","cited_by_count":52},{"id":"2010.12859","title":"Stable ResNet","created":"2020-10-24","cited_by_count":49},{"id":"2306.17759","title":"The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit","created":"2023-06-30","cited_by_count":47},{"id":"2302.10322","title":"Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation","created":"2023-02-20","cited_by_count":45},{"id":"2206.02768","title":"The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization","created":"2022-06-06","cited_by_count":41},{"id":"1906.02341","title":"How to Initialize your Network? Robust Initialization for WeightNorm & ResNets","created":"2019-06-05","cited_by_count":36},{"id":"2205.15242","title":"Re-parameterizing Your Optimizers rather than Architectures","created":"2022-05-30","cited_by_count":35},{"id":"2203.08120","title":"Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers","created":"2022-03-15","cited_by_count":28},{"id":"1809.08848","title":"Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function","created":"2018-09-24","cited_by_count":22},{"id":"2302.00453","title":"Width and Depth Limits Commute in Residual Networks","created":"2023-02-01","cited_by_count":21},{"id":"1911.09483","title":"MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning","created":"2019-11-17","cited_by_count":19},{"id":"2309.11197","title":"The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute","created":"2023-09-20","cited_by_count":10},{"id":"1911.03179","title":"Lipschitz Constrained Parameter Initialization for Deep Transformers","created":"2019-11-08","cited_by_count":9},{"id":"2310.02012","title":"Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion","created":"2023-10-03","cited_by_count":8},{"id":"2306.01610","title":"Centered Self-Attention Layers","created":"2023-06-02","cited_by_count":7},{"id":"2202.12411","title":"TrimBERT: Tailoring BERT for Trade-offs","created":"2022-02-24","cited_by_count":3}],"next_cursor":null}