{"list":"other","total":81,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2209.11895","title":"In-context Learning and Induction Heads","created":"2022-09-24","cited_by_count":964},{"id":"2301.05217","title":"Progress measures for grokking via mechanistic interpretability","created":"2023-01-12","cited_by_count":839},{"id":"1312.6120","title":"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks","created":"2013-12-20","cited_by_count":763},{"id":"2212.07677","title":"Transformers learn in-context by gradient descent","created":"2022-12-15","cited_by_count":732},{"id":"2208.01066","title":"What Can Transformers Learn In-Context? A Case Study of Simple Function Classes","created":"2022-08-01","cited_by_count":715},{"id":"2211.15661","title":"What learning algorithm is in-context learning? Investigations with linear models","created":"2022-11-28","cited_by_count":684},{"id":"1812.07956","title":"On Lazy Training in Differentiable Programming","created":"2018-12-19","cited_by_count":675},{"id":"2102.11174","title":"Linear Transformers Are Secretly Fast Weight Programmers","created":"2021-02-22","cited_by_count":502},{"id":"2306.09927","title":"Trained Transformers Learn Linear Models In-Context","created":"2023-06-16","cited_by_count":337},{"id":"2205.05055","title":"Data Distributional Properties Drive Emergent In-Context Learning in Transformers","created":"2022-04-22","cited_by_count":309},{"id":"2002.09277","title":"Kernel and Rich Regimes in Overparametrized Models","created":"2020-02-20","cited_by_count":306},{"id":"2306.04637","title":"Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection","created":"2023-06-07","cited_by_count":298},{"id":"1802.06509","title":"On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization","created":"2018-02-19","cited_by_count":284},{"id":"2306.00297","title":"Transformers learn to implement preconditioned gradient descent for in-context learning","created":"2023-06-01","cited_by_count":279},{"id":"1810.10531","title":"A mathematical theory of semantic development in deep neural networks","created":"2018-10-23","cited_by_count":186},{"id":"1810.02281","title":"A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks","created":"2018-10-04","cited_by_count":184},{"id":"1810.02032","title":"Gradient descent aligns the layers of deep linear networks","created":"2018-10-04","cited_by_count":182},{"id":"1806.00900","title":"Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced","created":"2018-06-04","cited_by_count":177},{"id":"2307.03576","title":"One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention","created":"2023-07-07","cited_by_count":170},{"id":"2309.07311","title":"Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs","created":"2023-09-13","cited_by_count":124},{"id":"2312.03002","title":"The mechanistic basis of data dependence and abrupt learning in an in-context classification task","created":"2023-12-03","cited_by_count":118},{"id":"2402.11004","title":"The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains","created":"2024-02-16","cited_by_count":118},{"id":"2402.14735","title":"How Transformers Learn Causal Structure with Gradient Descent","created":"2024-02-22","cited_by_count":118},{"id":"2310.05249","title":"In-Context Convergence of Transformers","created":"2023-10-08","cited_by_count":115},{"id":"2111.00034","title":"Neural Networks as Kernel Learners: The Silent Alignment Effect","created":"2021-10-29","cited_by_count":106},{"id":"2305.16380","title":"Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer","created":"2023-05-25","cited_by_count":102},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"2310.08391","title":"How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?","created":"2023-10-12","cited_by_count":96},{"id":"2210.09221","title":"Vision Transformers provably learn spatial structure","created":"2022-10-13","cited_by_count":93},{"id":"2404.07129","title":"What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation","created":"2024-04-10","cited_by_count":89},{"id":"2212.09803","title":"Training Trajectories of Language Models Across Scales","created":"2022-12-19","cited_by_count":81},{"id":"2311.08360","title":"The Transient Nature of Emergent In-Context Learning in Transformers","created":"2023-11-14","cited_by_count":76},{"id":"2308.16898","title":"Transformers as Support Vector Machines","created":"2023-08-31","cited_by_count":74},{"id":"2402.19442","title":"Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality","created":"2024-02-29","cited_by_count":74},{"id":"2306.13596","title":"Max-Margin Token Selection in Attention Mechanism","created":"2023-06-23","cited_by_count":58},{"id":"2405.11751","title":"Asymptotic theory of in-context learning by linear attention","created":"2024-05-20","cited_by_count":55},{"id":"2412.01003","title":"Competition Dynamics Shape Algorithmic Phases of In-Context Learning","created":"2024-12-01","cited_by_count":54},{"id":"2306.07042","title":"Transformers learn through gradual rank increase","created":"2023-06-12","cited_by_count":52},{"id":"2409.10559","title":"Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers","created":"2024-09-09","cited_by_count":50},{"id":"2310.17086","title":"Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression","created":"2023-10-26","cited_by_count":49},{"id":"2402.11867","title":"LoRA Training in the NTK Regime has No Spurious Local Minima","created":"2024-02-19","cited_by_count":48},{"id":"2402.01258","title":"Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape","created":"2024-02-02","cited_by_count":46},{"id":"2410.06833","title":"Dynamic metastability in the self-attention model","created":"2024-10-09","cited_by_count":46},{"id":"2406.02550","title":"Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks","created":"2024-06-04","cited_by_count":40},{"id":"2405.15712","title":"Infinite Limits of Multi-head Transformer Dynamics","created":"2024-05-24","cited_by_count":39},{"id":"2412.03782","title":"The broader spectrum of in-context learning","created":"2024-12-05","cited_by_count":35},{"id":"2402.13512","title":"From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers","created":"2024-02-21","cited_by_count":34},{"id":"2402.14951","title":"In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization","created":"2024-02-22","cited_by_count":33},{"id":"2402.05738","title":"Implicit Bias and Fast Convergence Rates for Self-attention","created":"2024-02-08","cited_by_count":31},{"id":"2402.14180","title":"Linear Transformers are Versatile In-Context Learners","created":"2024-02-21","cited_by_count":28},{"id":"2108.01538","title":"Geometry of Linear Convolutional Networks","created":"2021-08-03","cited_by_count":27},{"id":"2310.09753","title":"When can transformers reason with abstract symbols?","created":"2023-10-15","cited_by_count":24},{"id":"2407.10005","title":"Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond","created":"2024-07-13","cited_by_count":24},{"id":"2410.19637","title":"A distributional simplicity bias in the learning dynamics of transformers","created":"2024-10-25","cited_by_count":24},{"id":"1809.08587","title":"Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks","created":"2018-09-23","cited_by_count":22},{"id":"2406.06893","title":"Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot","created":"2024-06-11","cited_by_count":22},{"id":"2402.04084","title":"Provably learning a multi-head attention layer","created":"2024-02-06","cited_by_count":21},{"id":"2412.00104","title":"Differential learning kinetics govern the transition from memorization to generalization during in-context learning","created":"2024-11-27","cited_by_count":21},{"id":"2304.05752","title":"Function Space and Critical Points of Linear Convolutional Networks","created":"2023-04-12","cited_by_count":19},{"id":"2403.08699","title":"Implicit Regularization of Gradient Flow on One-Layer Softmax Attention","created":"2024-03-13","cited_by_count":19},{"id":"2410.23042","title":"Toward Understanding In-context vs. In-weight Learning","created":"2024-10-30","cited_by_count":19},{"id":"2310.08049","title":"Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability","created":"2023-10-12","cited_by_count":17},{"id":"2409.19345","title":"Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization","created":"2024-09-28","cited_by_count":17},{"id":"2408.17221","title":"Geometry of Lightning Self-Attention: Identifiability and Dimension","created":"2024-08-30","cited_by_count":16},{"id":"2406.00053","title":"Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting","created":"2024-05-28","cited_by_count":15},{"id":"2410.01774","title":"Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context","created":"2024-10-02","cited_by_count":15},{"id":"2310.12956","title":"Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems","created":"2023-10-19","cited_by_count":14},{"id":"2405.15618","title":"MLPs Learn In-Context on Regression and Classification Tasks","created":"2024-05-24","cited_by_count":13},{"id":"2409.17335","title":"Non-asymptotic Convergence of Training Transformers for Next-token Prediction","created":"2024-09-25","cited_by_count":13},{"id":"2406.03072","title":"Local to Global: Learning Dynamics and Effect of Initialization for Transformers","created":"2024-06-05","cited_by_count":11},{"id":"2410.21698","title":"On the Role of Depth and Looping for In-Context Learning with Task Diversity","created":"2024-10-29","cited_by_count":11},{"id":"2410.22244","title":"Abrupt Learning in Transformers: A Case Study on Matrix Completion","created":"2024-10-29","cited_by_count":11},{"id":"2411.07538","title":"Unraveling the Gradient Descent Dynamics of Transformers","created":"2024-11-12","cited_by_count":11},{"id":"2410.14581","title":"Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection","created":"2024-10-18","cited_by_count":9},{"id":"2410.23438","title":"Learning and Transferring Sparse Contextual Bigrams with Linear Transformers","created":"2024-10-30","cited_by_count":9},{"id":"2403.02233","title":"A Theoretical Analysis of Self-Supervised Learning for Vision Transformers","created":"2024-03-04","cited_by_count":6},{"id":"2410.12783","title":"Context-Scaling versus Task-Scaling in In-Context Learning","created":"2024-10-16","cited_by_count":6},{"id":"2309.13736","title":"Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups","created":"2023-09-24","cited_by_count":5},{"id":"2410.10101","title":"Learning Linear Attention in Polynomial Time","created":"2024-10-14","cited_by_count":5}],"next_cursor":null}