{"total":86,"rows":[{"id":"2305.07759","title":"TinyStories: How Small Can Language Models Be and Still Speak Coherent English?","created":"2023-05-12","cited_by_count":449,"area":null},{"id":"2309.16042","title":"Towards Best Practices of Activation Patching in Language Models: Metrics and Methods","created":"2023-09-27","cited_by_count":359,"area":"mechinterp"},{"id":"2306.09927","title":"Trained Transformers Learn Linear Models In-Context","created":"2023-06-16","cited_by_count":337,"area":null},{"id":"2306.00802","title":"Birth of a Transformer: A Memory Viewpoint","created":"2023-06-01","cited_by_count":161,"area":null},{"id":"2307.14430","title":"Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models","created":"2023-07-26","cited_by_count":121,"area":null},{"id":"2305.16380","title":"Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer","created":"2023-05-25","cited_by_count":102,"area":null},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98,"area":null},{"id":"2310.08391","title":"How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?","created":"2023-10-12","cited_by_count":96,"area":null},{"id":"2306.00946","title":"Exposing Attention Glitches with Flip-Flop Language Modeling","created":"2023-06-01","cited_by_count":75,"area":null},{"id":"2310.00535","title":"JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention","created":"2023-10-01","cited_by_count":52,"area":null},{"id":"2304.10411","title":"Attention Scheme Inspired Softmax Regression","created":"2023-04-20","cited_by_count":49,"area":null},{"id":"2303.16504","title":"An Over-parameterized Exponential Regression","created":"2023-03-29","cited_by_count":45,"area":null},{"id":"2405.19592","title":"Why Larger Language Models Do In-context Learning Differently?","created":"2024-05-30","cited_by_count":45,"area":null},{"id":"2401.15530","title":"An Information-Theoretic Analysis of In-Context Learning","created":"2024-01-28","cited_by_count":40,"area":null},{"id":"2402.04161","title":"Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains","created":"2024-02-06","cited_by_count":40,"area":null},{"id":"2504.10957","title":"When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers","created":"2025-04-15","cited_by_count":38,"area":"safety"},{"id":"2310.18362","title":"SoK: Memorization in General-Purpose Large Language Models","created":"2023-10-24","cited_by_count":37,"area":null},{"id":"2305.00660","title":"An Iterative Algorithm for Rescaled Hyperbolic Functions Regression","created":"2023-05-01","cited_by_count":35,"area":null},{"id":"2402.15607","title":"How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?","created":"2024-02-23","cited_by_count":35,"area":null},{"id":"2403.08081","title":"Mechanics of Next Token Prediction with Self-Attention","created":"2024-03-12","cited_by_count":34,"area":null},{"id":"2307.02419","title":"In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trick","created":"2023-07-05","cited_by_count":32,"area":null},{"id":"2309.07418","title":"A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time","created":"2023-09-14","cited_by_count":30,"area":null},{"id":"2312.01429","title":"Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars","created":"2023-12-03","cited_by_count":30,"area":null},{"id":"2503.10996","title":"Taming Knowledge Conflicts in Language Models","created":"2025-03-14","cited_by_count":30,"area":null},{"id":"2305.15287","title":"The Crucial Role of Normalization in Sharpness-Aware Minimization","created":"2023-05-24","cited_by_count":26,"area":null},{"id":"2402.11639","title":"In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness","created":"2024-02-18","cited_by_count":22,"area":null},{"id":"2408.13233","title":"Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time","created":"2024-08-23","cited_by_count":22,"area":null},{"id":"2410.11268","title":"Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent","created":"2024-10-15","cited_by_count":22,"area":null},{"id":"2403.08699","title":"Implicit Regularization of Gradient Flow on One-Layer Softmax Attention","created":"2024-03-13","cited_by_count":19,"area":null},{"id":"2410.01537","title":"Attention layers provably solve single-location regression","created":"2024-10-02","cited_by_count":17,"area":null},{"id":"2307.08352","title":"Zero-th Order Algorithm for Softmax Attention Optimization","created":"2023-07-17","cited_by_count":16,"area":null},{"id":"2411.10830","title":"One-Layer Transformer Provably Learns One-Nearest Neighbor In Context","created":"2024-11-16","cited_by_count":16,"area":null},{"id":"2412.17626","title":"Tracking the Feature Dynamics in LLM Training: A Mechanistic Study","created":"2024-12-23","cited_by_count":16,"area":"mechinterp"},{"id":"2406.18400","title":"Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers","created":"2024-06-26","cited_by_count":15,"area":null},{"id":"2410.11778","title":"On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures","created":"2024-10-15","cited_by_count":15,"area":null},{"id":"2504.12691","title":"Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations","created":"2025-04-17","cited_by_count":15,"area":null},{"id":"2406.01977","title":"What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding","created":"2024-06-04","cited_by_count":14,"area":null},{"id":"2408.04532","title":"How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression","created":"2024-08-08","cited_by_count":14,"area":null},{"id":"2410.02167","title":"Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis","created":"2024-10-03","cited_by_count":14,"area":"lr"},{"id":"2505.18651","title":"On the Emergence of Linear Analogies in Word Embeddings","created":"2025-05-24","cited_by_count":13,"area":null},{"id":"2310.12462","title":"Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights","created":"2023-10-19","cited_by_count":12,"area":null},{"id":"2502.12123","title":"On the Query Complexity of Verifier-Assisted Language Generation","created":"2025-02-17","cited_by_count":12,"area":"lr"},{"id":"2402.03485","title":"Attention Meets Post-hoc Interpretability: A Mathematical Perspective","created":"2024-02-05","cited_by_count":11,"area":null},{"id":"2406.03707","title":"What Should Embeddings Embed? Autoregressive Models Represent Latent Generating Distributions","created":"2024-06-06","cited_by_count":10,"area":null},{"id":"2510.06954","title":"From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics","created":"2025-10-08","cited_by_count":10,"area":null},{"id":"2311.15390","title":"Local Convergence of Approximate Newton Method for Two Layer Nonlinear Regression","created":"2023-11-26","cited_by_count":9,"area":null},{"id":"2410.02247","title":"Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization","created":"2024-10-03","cited_by_count":9,"area":null},{"id":"2410.09605","title":"Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis","created":"2024-10-12","cited_by_count":9,"area":null},{"id":"2304.06594","title":"Solving Tensor Low Cycle Rank Approximation","created":"2023-04-13","cited_by_count":8,"area":null},{"id":"2406.17167","title":"Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis","created":"2024-06-24","cited_by_count":8,"area":null},{"id":"2502.09863","title":"Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models","created":"2025-02-14","cited_by_count":8,"area":null},{"id":"2312.14226","title":"Deep de Finetti: Recovering Topic Distributions from Large Language Models","created":"2023-12-21","cited_by_count":7,"area":null},{"id":"2406.00944","title":"A Theory for Token-Level Harmonization in Retrieval-Augmented Generation","created":"2024-06-03","cited_by_count":7,"area":"rag"},{"id":"2406.02585","title":"Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task","created":"2024-05-30","cited_by_count":7,"area":null},{"id":"2311.13624","title":"A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer","created":"2023-11-22","cited_by_count":6,"area":null},{"id":"2311.14652","title":"One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space","created":"2023-11-24","cited_by_count":6,"area":null},{"id":"2407.21046","title":"Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines","created":"2024-07-22","cited_by_count":6,"area":null},{"id":"2512.18634","title":"From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers","created":"2025-12-21","cited_by_count":6,"area":null},{"id":"2405.16845","title":"On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability","created":"2024-05-27","cited_by_count":5,"area":null},{"id":"2410.23610","title":"Global Convergence in Training Large-Scale Transformers","created":"2024-10-31","cited_by_count":5,"area":null},{"id":"2502.06007","title":"Transformers versus the EM Algorithm in Multi-class Clustering","created":"2025-02-09","cited_by_count":5,"area":null},{"id":"2603.01514","title":"Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning","created":"2026-03-02","cited_by_count":5,"area":null},{"id":"2507.20443","title":"Provable In-Context Learning of Nonlinear Regression with Transformers","created":"2025-07-28","cited_by_count":4,"area":null},{"id":"2605.14075","title":"Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity","created":"2026-05-13","cited_by_count":4,"area":"lr"},{"id":"2312.00290","title":"Learning to forecast diagnostic parameters using pre-trained weather embedding","created":"2023-12-01","cited_by_count":3,"area":null},{"id":"2502.20681","title":"Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers","created":"2025-02-28","cited_by_count":3,"area":null},{"id":"2601.19208","title":"How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability","created":"2026-01-27","cited_by_count":3,"area":"mechinterp"},{"id":"2501.06382","title":"Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention","created":"2025-01-10","cited_by_count":2,"area":null},{"id":"2508.00127","title":"Structured Transformations for Stable and Interpretable Neural Computation","created":"2025-07-31","cited_by_count":2,"area":null},{"id":"2508.03649","title":"Cross-Model Semantics in Representation Learning","created":"2025-08-05","cited_by_count":2,"area":null},{"id":"2511.23239","title":"Towards Understanding Transformers in Learning Random Walks","created":"2025-11-28","cited_by_count":2,"area":null},{"id":"2602.19143","title":"Incremental Learning of Sparse Attention Patterns in Transformers","created":"2026-02-22","cited_by_count":2,"area":null},{"id":"2603.13381","title":"Beyond Linearity in Attention Projections: The Case for Nonlinear Queries","created":"2026-03-11","cited_by_count":2,"area":null},{"id":"2307.04057","title":"Bidirectional Attention as a Mixture of Continuous Word Experts","created":"2023-07-08","cited_by_count":1,"area":"moe"},{"id":"2508.02126","title":"Understanding Learning Dynamics Through Structured Representations","created":"2025-08-04","cited_by_count":1,"area":null},{"id":"2604.10848","title":"Transformers Learn Latent Mixture Models In-Context via Mirror Descent","created":"2026-04-12","cited_by_count":1,"area":null},{"id":"2605.01199","title":"Focus and Dilution: The Multi-stage Learning Process of Attention","created":"2026-05-02","cited_by_count":1,"area":null},{"id":"2608.06776","title":"Faster Query-Key Learning Sharpens Attention in Self-Attention Models","created":"2026-08-07","cited_by_count":1,"area":null},{"id":"2502.04580","title":"Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context","created":"2025-02-07","cited_by_count":0,"area":null},{"id":"2510.18638","title":"Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions","created":"2025-10-21","cited_by_count":0,"area":null},{"id":"2510.22594","title":"A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning","created":"2025-10-26","cited_by_count":0,"area":null},{"id":"2510.27004","title":"Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems","created":"2025-10-30","cited_by_count":0,"area":"moe"},{"id":"2602.02133","title":"A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse","created":"2026-02-02","cited_by_count":0,"area":"diffusion"},{"id":"2603.13459","title":"Reconciling In-Context and In-Weight Learning via Dual Representation Space Encoding","created":"2026-03-13","cited_by_count":0,"area":null},{"id":"2605.12697","title":"A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention","created":"2026-05-12","cited_by_count":0,"area":null},{"id":"2605.26489","title":"The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training","created":"2026-05-26","cited_by_count":0,"area":null}],"next_cursor":null}