{"total":102,"rows":[{"id":"2211.15661","title":"What learning algorithm is in-context learning? Investigations with linear models","created":"2022-11-28","cited_by_count":684,"area":null},{"id":"2306.04637","title":"Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection","created":"2023-06-07","cited_by_count":298,"area":null},{"id":"2305.15408","title":"Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective","created":"2023-05-24","cited_by_count":294,"area":"lr"},{"id":"2306.00297","title":"Transformers learn to implement preconditioned gradient descent for in-context learning","created":"2023-06-01","cited_by_count":279,"area":null},{"id":"2301.13196","title":"Looped Transformers as Programmable Computers","created":"2023-01-30","cited_by_count":237,"area":null},{"id":"2110.10090","title":"Inductive Biases and Variable Creation in Self-Attention Mechanisms","created":"2021-10-19","cited_by_count":144,"area":null},{"id":"2306.02896","title":"Representational Strengths and Limitations of Transformers","created":"2023-06-05","cited_by_count":133,"area":null},{"id":"2311.00208","title":"What Formal Languages Can Transformers Express? A Survey","created":"2023-11-01","cited_by_count":124,"area":null},{"id":"2307.03381","title":"Teaching Arithmetic to Small Transformers","created":"2023-07-07","cited_by_count":123,"area":null},{"id":"2305.19420","title":"What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization","created":"2023-05-30","cited_by_count":118,"area":null},{"id":"2301.05062","title":"Tracr: Compiled Transformers as a Laboratory for Interpretability","created":"2023-01-12","cited_by_count":104,"area":null},{"id":"2305.16380","title":"Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer","created":"2023-05-25","cited_by_count":102,"area":null},{"id":"2210.09221","title":"Vision Transformers provably learn spatial structure","created":"2022-10-13","cited_by_count":93,"area":null},{"id":"2303.04245","title":"How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding","created":"2023-03-07","cited_by_count":86,"area":null},{"id":"2210.14199","title":"Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models","created":"2022-10-25","cited_by_count":84,"area":null},{"id":"2310.08566","title":"Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining","created":"2023-10-12","cited_by_count":83,"area":"offlinerl"},{"id":"2402.09268","title":"Transformers, parallel computation, and logarithmic depth","created":"2024-02-14","cited_by_count":79,"area":null},{"id":"2302.06015","title":"A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity","created":"2023-02-12","cited_by_count":77,"area":null},{"id":"2306.00946","title":"Exposing Attention Glitches with Flip-Flop Language Modeling","created":"2023-06-01","cited_by_count":75,"area":null},{"id":"2309.06979","title":"Auto-Regressive Next-Token Predictors are Universal Learners","created":"2023-09-13","cited_by_count":67,"area":"lr"},{"id":"2405.18512","title":"Understanding Transformer Reasoning Capabilities via Graph Algorithms","created":"2024-05-28","cited_by_count":67,"area":"lr"},{"id":"2410.08292","title":"Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?","created":"2024-10-10","cited_by_count":65,"area":null},{"id":"2312.06528","title":"Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context","created":"2023-12-11","cited_by_count":63,"area":null},{"id":"2303.08117","title":"Do Transformers Parse while Predicting the Masked Word?","created":"2023-03-14","cited_by_count":48,"area":null},{"id":"2402.11867","title":"LoRA Training in the NTK Regime has No Spurious Local Minima","created":"2024-02-19","cited_by_count":48,"area":null},{"id":"2301.04589","title":"Memory Augmented Large Language Models are Computationally Universal","created":"2023-01-10","cited_by_count":47,"area":null},{"id":"2303.16504","title":"An Over-parameterized Exponential Regression","created":"2023-03-29","cited_by_count":45,"area":null},{"id":"2303.15725","title":"Solving Regularized Exp, Cosh and Sinh Regression Problems","created":"2023-03-28","cited_by_count":44,"area":null},{"id":"2304.13276","title":"The Closeness of In-Context Learning and Weight Shifting for Softmax Regression","created":"2023-04-26","cited_by_count":41,"area":null},{"id":"2307.11353","title":"What can a Single Attention Layer Learn? A Study Through the Random Features Lens","created":"2023-07-21","cited_by_count":40,"area":null},{"id":"2402.04161","title":"Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains","created":"2024-02-06","cited_by_count":40,"area":null},{"id":"2307.11007","title":"Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization","created":"2023-07-20","cited_by_count":39,"area":null},{"id":"2309.11420","title":"Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models","created":"2023-09-20","cited_by_count":39,"area":"diffusion"},{"id":"2305.18699","title":"Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input","created":"2023-05-30","cited_by_count":32,"area":null},{"id":"2305.18787","title":"Universality and Limitations of Prompt Tuning","created":"2023-05-30","cited_by_count":30,"area":null},{"id":"2312.01429","title":"Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars","created":"2023-12-03","cited_by_count":30,"area":null},{"id":"2402.01107","title":"Simulation of Graph Algorithms with Looped Transformers","created":"2024-02-02","cited_by_count":30,"area":null},{"id":"2411.06646","title":"Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data","created":"2024-11-11","cited_by_count":30,"area":null},{"id":"2306.02295","title":"A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Models","created":"2023-06-04","cited_by_count":29,"area":null},{"id":"2310.13088","title":"Sequence Length Independent Norm-Based Generalization Bounds for Transformers","created":"2023-10-19","cited_by_count":29,"area":null},{"id":"2405.04669","title":"Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics","created":"2024-05-07","cited_by_count":27,"area":"lr"},{"id":"2407.17686","title":"Transformers on Markov Data: Constant Depth Suffices","created":"2024-07-25","cited_by_count":27,"area":null},{"id":"2407.03310","title":"Universal Length Generalization with Turing Programs","created":"2024-07-03","cited_by_count":25,"area":null},{"id":"2407.15160","title":"When Can Transformers Count to n?","created":"2024-07-21","cited_by_count":24,"area":null},{"id":"2503.07932","title":"A Theory of Learning with Autoregressive Chain of Thought","created":"2025-03-11","cited_by_count":23,"area":"lr"},{"id":"2402.04084","title":"Provably learning a multi-head attention layer","created":"2024-02-06","cited_by_count":21,"area":null},{"id":"2402.13934","title":"Do Efficient Transformers Really Save Computation?","created":"2024-02-21","cited_by_count":20,"area":null},{"id":"2306.13945","title":"Large Sequence Models for Sequential Decision-Making: A Survey","created":"2023-06-24","cited_by_count":19,"area":null},{"id":"2410.09375","title":"Looped ReLU MLPs May Be All You Need as Practical Programmable Computers","created":"2024-10-12","cited_by_count":19,"area":null},{"id":"2307.01189","title":"Trainable Transformer in Transformer","created":"2023-07-03","cited_by_count":17,"area":null},{"id":"2404.18444","title":"U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models","created":"2024-04-29","cited_by_count":17,"area":"diffusion"},{"id":"2212.14852","title":"An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models","created":"2022-12-30","cited_by_count":16,"area":null},{"id":"2310.11685","title":"Why Softmax Attention Outperforms Linear Attention","created":"2023-10-18","cited_by_count":15,"area":null},{"id":"2401.09074","title":"Code Simulation Challenges for Large Language Models","created":"2024-01-17","cited_by_count":15,"area":null},{"id":"2410.03170","title":"Autoregressive Large Language Models are Computationally Universal","created":"2024-10-04","cited_by_count":15,"area":null},{"id":"2401.12947","title":"Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion","created":"2024-01-23","cited_by_count":14,"area":null},{"id":"2410.05603","title":"Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition","created":"2024-10-08","cited_by_count":14,"area":"mechinterp"},{"id":"2506.05295","title":"Sample Complexity and Representation Ability of Test-time Scaling Paradigms","created":"2025-06-05","cited_by_count":12,"area":"lr"},{"id":"2502.09844","title":"Solving Empirical Bayes via Transformers","created":"2025-02-14","cited_by_count":11,"area":null},{"id":"2503.01805","title":"Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers","created":"2025-03-03","cited_by_count":11,"area":null},{"id":"2410.04271","title":"Fundamental Limitations on Subquadratic Alternatives to Transformers","created":"2024-10-05","cited_by_count":9,"area":"ssm"},{"id":"2410.09605","title":"Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis","created":"2024-10-12","cited_by_count":9,"area":null},{"id":"2406.04090","title":"Interpretable Lightweight Transformer via Unrolling of Learned Graph Smoothness Priors","created":"2024-06-06","cited_by_count":8,"area":null},{"id":"2410.11474","title":"How Transformers Get Rich: Approximation and Dynamics Analysis","created":"2024-10-15","cited_by_count":8,"area":null},{"id":"2310.20051","title":"The Expressibility of Polynomial based Attention Scheme","created":"2023-10-30","cited_by_count":7,"area":null},{"id":"2501.18187","title":"On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning","created":"2025-01-30","cited_by_count":7,"area":null},{"id":"2502.03568","title":"Code Simulation as a Proxy for High-order Tasks in Large Language Models","created":"2025-02-05","cited_by_count":7,"area":null},{"id":"2407.21046","title":"Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines","created":"2024-07-22","cited_by_count":6,"area":null},{"id":"2503.01544","title":"Compositional Reasoning with Transformers, RNNs, and Chain of Thought","created":"2025-03-03","cited_by_count":6,"area":"lr"},{"id":"2407.16153","title":"On the Benefits of Rank in Attention Layers","created":"2024-07-23","cited_by_count":5,"area":null},{"id":"2410.10101","title":"Learning Linear Attention in Polynomial Time","created":"2024-10-14","cited_by_count":5,"area":null},{"id":"2504.13558","title":"Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective","created":"2025-04-18","cited_by_count":5,"area":null},{"id":"2505.03205","title":"Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insights","created":"2025-05-06","cited_by_count":5,"area":null},{"id":"2303.17655","title":"Q-Learning based system for path planning with unmanned aerial vehicles swarms in obstacle environments","created":"2023-03-30","cited_by_count":4,"area":null},{"id":"2403.16097","title":"Can Language Models Pretend Solvers? Logic Code Simulation with LLMs","created":"2024-03-24","cited_by_count":4,"area":null},{"id":"2410.16401","title":"Simplicity Bias via Global Convergence of Sharpness Minimization","created":"2024-10-21","cited_by_count":4,"area":null},{"id":"2502.15609","title":"On the Robustness of Transformers against Context Hijacking for Linear Classification","created":"2025-02-21","cited_by_count":4,"area":null},{"id":"2502.16763","title":"Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networks","created":"2025-02-24","cited_by_count":4,"area":null},{"id":"2511.03548","title":"Flat Minima and Generalization: Insights from Stochastic Convex Optimization","created":"2025-11-05","cited_by_count":4,"area":null},{"id":"2210.06741","title":"Why self-attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries","created":"2022-10-13","cited_by_count":3,"area":null},{"id":"2409.12446","title":"Neural Networks Generalize on Low Complexity Data","created":"2024-09-19","cited_by_count":3,"area":null},{"id":"2501.15857","title":"Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data?","created":"2025-01-27","cited_by_count":3,"area":null},{"id":"2503.11272","title":"When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective","created":"2025-03-14","cited_by_count":3,"area":null},{"id":"2509.09001","title":"Fast attention mechanisms: a tale of parallelism","created":"2025-09-10","cited_by_count":3,"area":null},{"id":"2209.00735","title":"Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms","created":"2022-09-01","cited_by_count":2,"area":null},{"id":"2506.23551","title":"A unified framework for establishing the universal approximation of transformer-type architectures","created":"2025-06-30","cited_by_count":2,"area":null},{"id":"2509.10414","title":"Is In-Context Learning Learning?","created":"2025-09-12","cited_by_count":2,"area":null},{"id":"2512.09275","title":"Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression","created":"2025-12-10","cited_by_count":2,"area":null},{"id":"2605.08811","title":"Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity","created":"2026-05-09","cited_by_count":2,"area":null},{"id":"2605.22471","title":"Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers","created":"2026-05-21","cited_by_count":2,"area":null},{"id":"2405.06003","title":"Binary Hypothesis Testing for Softmax Models and Leverage Score Models","created":"2024-05-09","cited_by_count":1,"area":null},{"id":"2407.10949","title":"Representing Rule-based Chatbots with Transformers","created":"2024-07-15","cited_by_count":1,"area":null},{"id":"2506.18739","title":"On the Existence of Universal Simulators of Attention","created":"2025-06-23","cited_by_count":1,"area":null},{"id":"2510.03678","title":"Towards Sampling Data Structures for Tensor Products in Turnstile Streams","created":"2025-10-04","cited_by_count":1,"area":null},{"id":"2601.08061","title":"Universal computation is intrinsic to language model decoding","created":"2026-01-12","cited_by_count":1,"area":null},{"id":"2602.01763","title":"A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention","created":"2026-02-02","cited_by_count":1,"area":null},{"id":"2604.14727","title":"Geometric Capacity of Transformers: A Tropical Geometry Perspective","created":"2026-04-16","cited_by_count":1,"area":null},{"id":"2604.25800","title":"Barriers to Universal Reasoning With Transformers (And How to Overcome Them)","created":"2026-04-28","cited_by_count":1,"area":"lr"},{"id":"2605.31423","title":"Fixed Universal Transformers","created":"2026-05-29","cited_by_count":1,"area":null},{"id":"2605.08150","title":"A PyTorch Library of Turing-Complete Neural Networks","created":"2026-05-03","cited_by_count":0,"area":null}],"next_cursor":"100"}