{"list":"other","total":67,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2010.11929","title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","created":"2020-10-22","cited_by_count":30186},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2104.14294","title":"Emerging Properties in Self-Supervised Vision Transformers","created":"2021-04-29","cited_by_count":6968},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"2202.05262","title":"Locating and Editing Factual Associations in GPT","created":"2022-02-10","cited_by_count":2691},{"id":"1806.07572","title":"Neural Tangent Kernel: Convergence and Generalization in Neural Networks","created":"2018-06-20","cited_by_count":2621},{"id":"2211.14730","title":"A Time Series is Worth 64 Words: Long-term Forecasting with Transformers","created":"2022-11-27","cited_by_count":2039},{"id":"1703.04730","title":"Understanding Black-box Predictions via Influence Functions","created":"2017-03-14","cited_by_count":1741},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"2010.01412","title":"Sharpness-Aware Minimization for Efficiently Improving Generalization","created":"2020-10-03","cited_by_count":1417},{"id":"2211.00593","title":"Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small","created":"2022-11-01","cited_by_count":1165},{"id":"2209.11895","title":"In-context Learning and Induction Heads","created":"2022-09-24","cited_by_count":964},{"id":"1702.03118","title":"Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning","created":"2017-02-10","cited_by_count":950},{"id":"2002.04745","title":"On Layer Normalization in the Transformer Architecture","created":"2020-02-12","cited_by_count":922},{"id":"2301.05217","title":"Progress measures for grokking via mechanistic interpretability","created":"2023-01-12","cited_by_count":839},{"id":"2304.14997","title":"Towards Automated Circuit Discovery for Mechanistic Interpretability","created":"2023-04-28","cited_by_count":724},{"id":"2302.05442","title":"Scaling Vision Transformers to 22 Billion Parameters","created":"2023-02-10","cited_by_count":694},{"id":"1812.07956","title":"On Lazy Training in Differentiable Programming","created":"2018-12-19","cited_by_count":675},{"id":"2201.02177","title":"Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets","created":"2022-01-06","cited_by_count":629},{"id":"2304.14767","title":"Dissecting Recall of Factual Associations in Auto-Regressive Language Models","created":"2023-04-28","cited_by_count":528},{"id":"2305.18654","title":"Faith and Fate: Limits of Transformers on Compositionality","created":"2023-05-29","cited_by_count":503},{"id":"1804.06561","title":"A Mean Field View of the Landscape of Two-Layers Neural Networks","created":"2018-04-18","cited_by_count":466},{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427},{"id":"2306.09927","title":"Trained Transformers Learn Linear Models In-Context","created":"2023-06-16","cited_by_count":337},{"id":"1904.01557","title":"Analysing Mathematical Reasoning Abilities of Neural Models","created":"2019-04-02","cited_by_count":335},{"id":"1406.2572","title":"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization","created":"2014-06-10","cited_by_count":289},{"id":"2306.00297","title":"Transformers learn to implement preconditioned gradient descent for in-context learning","created":"2023-06-01","cited_by_count":279},{"id":"1608.05859","title":"Using the Output Embedding to Improve Language Models","created":"2016-08-20","cited_by_count":272},{"id":"2205.10343","title":"Towards Understanding Grokking: An Effective Theory of Representation Learning","created":"2022-05-20","cited_by_count":230},{"id":"2210.12353","title":"Leveraging Large Language Models for Multiple Choice Question Answering","created":"2022-10-22","cited_by_count":203},{"id":"2207.08799","title":"Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit","created":"2022-07-18","cited_by_count":187},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"2307.03576","title":"One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention","created":"2023-07-07","cited_by_count":170},{"id":"2306.00802","title":"Birth of a Transformer: A Memory Viewpoint","created":"2023-06-01","cited_by_count":161},{"id":"1809.09573","title":"Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview","created":"2018-09-25","cited_by_count":160},{"id":"2202.08658","title":"The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks","created":"2022-02-17","cited_by_count":142},{"id":"2305.05465","title":"The emergence of clusters in self-attention dynamics","created":"2023-05-09","cited_by_count":140},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"2307.03381","title":"Teaching Arithmetic to Small Transformers","created":"2023-07-07","cited_by_count":123},{"id":"2210.06313","title":"The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers","created":"2022-10-12","cited_by_count":122},{"id":"2312.03002","title":"The mechanistic basis of data dependence and abrupt learning in an in-context classification task","created":"2023-12-03","cited_by_count":118},{"id":"2402.11004","title":"The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains","created":"2024-02-16","cited_by_count":118},{"id":"1906.02762","title":"Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View","created":"2019-06-06","cited_by_count":113},{"id":"2306.07304","title":"A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation","created":"2023-06-11","cited_by_count":109},{"id":"2305.16380","title":"Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer","created":"2023-05-25","cited_by_count":102},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"2310.04564","title":"ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models","created":"2023-10-06","cited_by_count":94},{"id":"2210.09221","title":"Vision Transformers provably learn spatial structure","created":"2022-10-13","cited_by_count":93},{"id":"1705.10412","title":"Gradient Descent Can Take Exponential Time to Escape Saddle Points","created":"2017-05-29","cited_by_count":74},{"id":"2410.01131","title":"nGPT: Normalized Transformer with Representation Learning on the Hypersphere","created":"2024-10-01","cited_by_count":70},{"id":"2111.04138","title":"Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis","created":"2021-11-07","cited_by_count":65},{"id":"2304.00488","title":"Saddle-to-Saddle Dynamics in Diagonal Linear Networks","created":"2023-04-02","cited_by_count":57},{"id":"2306.07042","title":"Transformers learn through gradual rank increase","created":"2023-06-12","cited_by_count":52},{"id":"2312.16903","title":"Spike No More: Stabilizing the Pre-training of Large Language Models","created":"2023-12-28","cited_by_count":52},{"id":"2312.14820","title":"How Smooth Is Attention?","created":"2023-12-22","cited_by_count":44},{"id":"2308.15594","title":"Learning the greatest common divisor: explaining transformer predictions","created":"2023-08-29","cited_by_count":33},{"id":"2406.02128","title":"Iteration Head: A Mechanistic Study of Chain-of-Thought","created":"2024-06-04","cited_by_count":31},{"id":"2306.16921","title":"Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputs","created":"2023-06-29","cited_by_count":22},{"id":"2211.00170","title":"What is my math transformer doing? -- Three results on interpretability and generalization","created":"2022-10-31","cited_by_count":19},{"id":"2306.06479","title":"Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs","created":"2023-06-10","cited_by_count":16},{"id":"2310.12956","title":"Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems","created":"2023-10-19","cited_by_count":14},{"id":"2405.16397","title":"AdaFisher: Adaptive Second Order Optimization via Fisher Information","created":"2024-05-26","cited_by_count":14},{"id":"2406.10834","title":"Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning","created":"2024-06-16","cited_by_count":13},{"id":"2310.14814","title":"Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Bias","created":"2023-10-23","cited_by_count":7},{"id":"2403.09889","title":"Generalization of Scaled Deep ResNets in the Mean-Field Regime","created":"2024-03-14","cited_by_count":3}],"next_cursor":null}