{"list":"other","total":55,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2212.07677","title":"Transformers learn in-context by gradient descent","created":"2022-12-15","cited_by_count":732},{"id":"2208.01066","title":"What Can Transformers Learn In-Context? A Case Study of Simple Function Classes","created":"2022-08-01","cited_by_count":715},{"id":"2211.15661","title":"What learning algorithm is in-context learning? Investigations with linear models","created":"2022-11-28","cited_by_count":684},{"id":"2306.09927","title":"Trained Transformers Learn Linear Models In-Context","created":"2023-06-16","cited_by_count":337},{"id":"2306.04637","title":"Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection","created":"2023-06-07","cited_by_count":298},{"id":"2306.00297","title":"Transformers learn to implement preconditioned gradient descent for in-context learning","created":"2023-06-01","cited_by_count":279},{"id":"1906.01820","title":"Risks from Learned Optimization in Advanced Machine Learning Systems","created":"2019-06-05","cited_by_count":258},{"id":"2212.10559","title":"Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers","created":"2022-12-20","cited_by_count":204},{"id":"2207.08799","title":"Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit","created":"2022-07-18","cited_by_count":187},{"id":"2307.03576","title":"One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention","created":"2023-07-07","cited_by_count":170},{"id":"2206.15144","title":"Neural Networks can Learn Representations with Gradient Descent","created":"2022-06-30","cited_by_count":161},{"id":"1904.12191","title":"Linearized two-layers neural networks in high dimension","created":"2019-04-27","cited_by_count":155},{"id":"2306.15063","title":"Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression","created":"2023-06-26","cited_by_count":146},{"id":"2302.11055","title":"SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics","created":"2023-02-21","cited_by_count":143},{"id":"1605.00251","title":"A vector-contraction inequality for Rademacher complexities","created":"2016-05-01","cited_by_count":142},{"id":"2202.08658","title":"The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks","created":"2022-02-17","cited_by_count":142},{"id":"2003.10409","title":"Online stochastic gradient descent on non-convex losses from high-dimensional inference","created":"2020-03-23","cited_by_count":119},{"id":"2305.19420","title":"What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization","created":"2023-05-30","cited_by_count":118},{"id":"2402.14735","title":"How Transformers Learn Causal Structure with Gradient Descent","created":"2024-02-22","cited_by_count":118},{"id":"1708.03395","title":"Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models","created":"2017-08-10","cited_by_count":116},{"id":"2310.05249","title":"In-Context Convergence of Transformers","created":"2023-10-08","cited_by_count":115},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"2310.08391","title":"How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?","created":"2023-10-12","cited_by_count":96},{"id":"2210.15651","title":"Learning Single-Index Models with Shallow Neural Networks","created":"2022-10-27","cited_by_count":92},{"id":"1910.01619","title":"Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks","created":"2019-10-03","cited_by_count":79},{"id":"2310.10616","title":"How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations","created":"2023-10-16","cited_by_count":75},{"id":"2402.19442","title":"Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality","created":"2024-02-29","cited_by_count":74},{"id":"2305.18270","title":"How Two-Layer Neural Networks Learn, One (Giant) Step at a Time","created":"2023-05-29","cited_by_count":68},{"id":"2410.08292","title":"Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?","created":"2024-10-10","cited_by_count":65},{"id":"2209.14863","title":"Neural Networks Efficiently Learn Low-Dimensional Representations with SGD","created":"2022-09-29","cited_by_count":63},{"id":"2312.06528","title":"Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context","created":"2023-12-11","cited_by_count":63},{"id":"2305.10633","title":"Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models","created":"2023-05-18","cited_by_count":61},{"id":"2303.00055","title":"Learning time-scales in two-layers neural networks","created":"2023-02-28","cited_by_count":57},{"id":"2310.19793","title":"On Learning Gaussian Multi-index Models with Gradient Flow","created":"2023-10-30","cited_by_count":57},{"id":"2405.11751","title":"Asymptotic theory of in-context learning by linear attention","created":"2024-05-20","cited_by_count":55},{"id":"2402.03220","title":"The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents","created":"2024-02-05","cited_by_count":52},{"id":"2409.10559","title":"Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers","created":"2024-09-09","cited_by_count":50},{"id":"2402.01258","title":"Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape","created":"2024-02-02","cited_by_count":46},{"id":"2406.01581","title":"Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit","created":"2024-06-03","cited_by_count":46},{"id":"2401.15530","title":"An Information-Theoretic Analysis of In-Context Learning","created":"2024-01-28","cited_by_count":40},{"id":"1708.05932","title":"Fundamental Limits of Weak Recovery with Applications to Phase Retrieval","created":"2017-08-20","cited_by_count":39},{"id":"2405.15459","title":"Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functions","created":"2024-05-24","cited_by_count":37},{"id":"2408.12186","title":"Transformers are Minimax Optimal Nonparametric In-Context Learners","created":"2024-08-22","cited_by_count":37},{"id":"2402.15607","title":"How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?","created":"2024-02-23","cited_by_count":35},{"id":"2004.13748","title":"Learning Polynomials of Few Relevant Dimensions","created":"2020-04-28","cited_by_count":34},{"id":"2402.14951","title":"In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization","created":"2024-02-22","cited_by_count":33},{"id":"2104.04244","title":"How rotational invariance of common kernels prevents generalization in high dimensions","created":"2021-04-09","cited_by_count":32},{"id":"2309.03843","title":"Gradient-Based Feature Learning under Structured Data","created":"2023-09-07","cited_by_count":32},{"id":"2306.16361","title":"Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time","created":"2023-06-28","cited_by_count":20},{"id":"2402.05787","title":"How do Transformers perform In-Context Autoregressive Learning?","created":"2024-02-08","cited_by_count":17},{"id":"2307.06887","title":"Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks","created":"2023-07-13","cited_by_count":16},{"id":"2407.05622","title":"On the Complexity of Learning Sparse Functions with Statistical and Gradient Queries","created":"2024-07-08","cited_by_count":15},{"id":"2406.11828","title":"Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinations","created":"2024-06-17","cited_by_count":13}],"next_cursor":null}