{"list":"other","total":51,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2312.00752","title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","created":"2023-12-01","cited_by_count":4718},{"id":"2104.09864","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","created":"2021-04-20","cited_by_count":4703},{"id":"1609.07843","title":"Pointer Sentinel Mixture Models","created":"2016-09-26","cited_by_count":3096},{"id":"1610.01644","title":"Understanding intermediate layers using linear classifier probes","created":"2016-10-05","cited_by_count":1353},{"id":"2108.12409","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","created":"2021-08-27","cited_by_count":1103},{"id":"2212.07677","title":"Transformers learn in-context by gradient descent","created":"2022-12-15","cited_by_count":732},{"id":"2208.01066","title":"What Can Transformers Learn In-Context? A Case Study of Simple Function Classes","created":"2022-08-01","cited_by_count":715},{"id":"2211.15661","title":"What learning algorithm is in-context learning? Investigations with linear models","created":"2022-11-28","cited_by_count":684},{"id":"1708.06633","title":"Nonparametric regression using deep neural networks with ReLU activation function","created":"2017-08-22","cited_by_count":433},{"id":"1912.10077","title":"Are Transformers universal approximators of sequence-to-sequence functions?","created":"2019-12-20","cited_by_count":406},{"id":"2306.09927","title":"Trained Transformers Learn Linear Models In-Context","created":"2023-06-16","cited_by_count":337},{"id":"2306.04637","title":"Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection","created":"2023-06-07","cited_by_count":298},{"id":"1906.06755","title":"Theoretical Limitations of Self-Attention in Neural Sequence Models","created":"2019-06-16","cited_by_count":288},{"id":"2306.00297","title":"Transformers learn to implement preconditioned gradient descent for in-context learning","created":"2023-06-01","cited_by_count":279},{"id":"2301.13196","title":"Looped Transformers as Programmable Computers","created":"2023-01-30","cited_by_count":237},{"id":"2402.01032","title":"Repeat After Me: Transformers are Better than State Space Models at Copying","created":"2024-02-01","cited_by_count":216},{"id":"2106.06981","title":"Thinking Like Transformers","created":"2021-06-13","cited_by_count":184},{"id":"1806.00900","title":"Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced","created":"2018-06-04","cited_by_count":177},{"id":"2307.03576","title":"One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention","created":"2023-07-07","cited_by_count":170},{"id":"2306.00802","title":"Birth of a Transformer: A Memory Viewpoint","created":"2023-06-01","cited_by_count":161},{"id":"2110.10090","title":"Inductive Biases and Variable Creation in Self-Attention Mechanisms","created":"2021-10-19","cited_by_count":144},{"id":"2106.16213","title":"Saturated Transformers are Constant-Depth Threshold Circuits","created":"2021-06-30","cited_by_count":139},{"id":"2312.03002","title":"The mechanistic basis of data dependence and abrupt learning in an in-context classification task","created":"2023-12-03","cited_by_count":118},{"id":"2402.11004","title":"The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains","created":"2024-02-16","cited_by_count":118},{"id":"2402.14735","title":"How Transformers Learn Causal Structure with Gradient Descent","created":"2024-02-22","cited_by_count":118},{"id":"2310.05249","title":"In-Context Convergence of Transformers","created":"2023-10-08","cited_by_count":115},{"id":"2107.13163","title":"Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers","created":"2021-07-28","cited_by_count":102},{"id":"2305.16380","title":"Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer","created":"2023-05-25","cited_by_count":102},{"id":"2009.10713","title":"Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't","created":"2020-09-22","cited_by_count":95},{"id":"2205.09921","title":"KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation","created":"2022-05-20","cited_by_count":86},{"id":"2402.09268","title":"Transformers, parallel computation, and logarithmic depth","created":"2024-02-14","cited_by_count":79},{"id":"1810.06397","title":"A Priori Estimates of the Population Risk for Two-layer Neural Networks","created":"2018-10-15","cited_by_count":76},{"id":"2308.16898","title":"Transformers as Support Vector Machines","created":"2023-08-31","cited_by_count":74},{"id":"2402.19442","title":"Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality","created":"2024-02-29","cited_by_count":74},{"id":"2306.13596","title":"Max-Margin Token Selection in Attention Mechanism","created":"2023-06-23","cited_by_count":58},{"id":"2310.19793","title":"On Learning Gaussian Multi-index Models with Gradient Flow","created":"2023-10-30","cited_by_count":57},{"id":"2306.07042","title":"Transformers learn through gradual rank increase","created":"2023-06-12","cited_by_count":52},{"id":"2409.10559","title":"Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers","created":"2024-09-09","cited_by_count":50},{"id":"2407.07011","title":"Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning","created":"2024-07-09","cited_by_count":46},{"id":"2408.14332","title":"One-layer transformers fail to solve the induction heads task","created":"2024-08-26","cited_by_count":32},{"id":"2402.05738","title":"Implicit Bias and Fast Convergence Rates for Self-attention","created":"2024-02-08","cited_by_count":31},{"id":"2408.09503","title":"Out-of-distribution generalization via composition: a lens through induction heads in Transformers","created":"2024-08-18","cited_by_count":28},{"id":"2407.17686","title":"Transformers on Markov Data: Constant Depth Suffices","created":"2024-07-25","cited_by_count":27},{"id":"2402.00522","title":"Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling","created":"2024-02-01","cited_by_count":25},{"id":"2406.06893","title":"Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot","created":"2024-06-11","cited_by_count":22},{"id":"2404.01601","title":"What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks","created":"2024-04-02","cited_by_count":19},{"id":"2112.02393","title":"Optimization-Based Separations for Neural Networks","created":"2021-12-04","cited_by_count":17},{"id":"2311.13774","title":"Learning Hierarchical Polynomials with Three-Layer Neural Networks","created":"2023-11-23","cited_by_count":9}],"next_cursor":null}