{"total":133,"rows":[{"id":"2306.14048","title":"H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models","created":"2023-06-24","cited_by_count":832,"area":"inference"},{"id":"2402.01032","title":"Repeat After Me: Transformers are Better than State Space Models at Copying","created":"2024-02-01","cited_by_count":216,"area":null},{"id":"2310.16028","title":"What Algorithms can Transformers Learn? A Study in Length Generalization","created":"2023-10-24","cited_by_count":199,"area":null},{"id":"2404.15758","title":"Let's Think Dot by Dot: Hidden Computation in Transformer Language Models","created":"2024-04-24","cited_by_count":191,"area":"lr"},{"id":"2311.00208","title":"What Formal Languages Can Transformers Express? A Survey","created":"2023-11-01","cited_by_count":124,"area":null},{"id":"2402.09268","title":"Transformers, parallel computation, and logarithmic depth","created":"2024-02-14","cited_by_count":79,"area":null},{"id":"2402.09963","title":"Why are Sensitive Functions Hard for Transformers?","created":"2024-02-15","cited_by_count":74,"area":null},{"id":"2405.18512","title":"Understanding Transformer Reasoning Capabilities via Graph Algorithms","created":"2024-05-28","cited_by_count":67,"area":"lr"},{"id":"2402.08871","title":"Position: Topological Deep Learning is the New Frontier for Relational Learning","created":"2024-02-14","cited_by_count":66,"area":null},{"id":"2402.18510","title":"RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval","created":"2024-02-28","cited_by_count":64,"area":null},{"id":"2404.16112","title":"Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges","created":"2024-04-24","cited_by_count":64,"area":"ssm"},{"id":"2402.08164","title":"On Limitations of the Transformer Architecture","created":"2024-02-13","cited_by_count":63,"area":null},{"id":"2405.17394","title":"The Expressive Capacity of State Space Models: A Formal Language Perspective","created":"2024-05-27","cited_by_count":46,"area":null},{"id":"2412.02975","title":"Theoretical limitations of multi-layer Transformer","created":"2024-12-04","cited_by_count":45,"area":null},{"id":"2310.04064","title":"How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation","created":"2023-10-06","cited_by_count":44,"area":null},{"id":"2401.15530","title":"An Information-Theoretic Analysis of In-Context Learning","created":"2024-01-28","cited_by_count":40,"area":null},{"id":"2310.12680","title":"On the Optimization and Generalization of Multi-head Attention","created":"2023-10-19","cited_by_count":37,"area":null},{"id":"2408.14332","title":"One-layer transformers fail to solve the induction heads task","created":"2024-08-26","cited_by_count":32,"area":null},{"id":"2402.05738","title":"Implicit Bias and Fast Convergence Rates for Self-attention","created":"2024-02-08","cited_by_count":31,"area":null},{"id":"2410.02338","title":"How Much Can RAG Help the Reasoning of LLM?","created":"2024-10-03","cited_by_count":31,"area":"lr"},{"id":"2306.04933","title":"InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding","created":"2023-06-08","cited_by_count":30,"area":null},{"id":"2309.07418","title":"A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time","created":"2023-09-14","cited_by_count":30,"area":null},{"id":"2405.16411","title":"Training Tensor Attention Efficiently: From Cubic to Almost Linear Time","created":"2024-05-26","cited_by_count":30,"area":null},{"id":"2406.09347","title":"Separations in the Representational Capabilities of Transformers and Recurrent Architectures","created":"2024-06-13","cited_by_count":29,"area":null},{"id":"2502.13063","title":"Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity","created":"2025-02-18","cited_by_count":28,"area":"inference"},{"id":"2407.17686","title":"Transformers on Markov Data: Constant Depth Suffices","created":"2024-07-25","cited_by_count":27,"area":null},{"id":"2308.12247","title":"How to Protect Copyright Data in Optimization of Large Language Models?","created":"2023-08-23","cited_by_count":26,"area":null},{"id":"2402.14753","title":"Prompting a Pretrained Transformer Can Be a Universal Approximator","created":"2024-02-22","cited_by_count":25,"area":null},{"id":"2406.13261","title":"BeHonest: Benchmarking Honesty in Large Language Models","created":"2024-06-19","cited_by_count":24,"area":null},{"id":"2407.15160","title":"When Can Transformers Count to n?","created":"2024-07-21","cited_by_count":24,"area":null},{"id":"2402.11639","title":"In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness","created":"2024-02-18","cited_by_count":22,"area":null},{"id":"2406.06893","title":"Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot","created":"2024-06-11","cited_by_count":22,"area":null},{"id":"2402.01929","title":"Sample, estimate, aggregate: A recipe for causal discovery foundation models","created":"2024-02-02","cited_by_count":21,"area":null},{"id":"2402.04084","title":"Provably learning a multi-head attention layer","created":"2024-02-06","cited_by_count":21,"area":null},{"id":"2504.14370","title":"Density Measures for Language Generation","created":"2025-04-19","cited_by_count":21,"area":null},{"id":"2402.13934","title":"Do Efficient Transformers Really Save Computation?","created":"2024-02-21","cited_by_count":20,"area":null},{"id":"2310.11685","title":"Why Softmax Attention Outperforms Linear Attention","created":"2023-10-18","cited_by_count":15,"area":null},{"id":"2401.15127","title":"Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness","created":"2024-01-26","cited_by_count":14,"area":null},{"id":"2511.05295","title":"Language Generation and Identification From Partial Enumeration: Tight Density Bounds and Topological Characterizations","created":"2025-11-07","cited_by_count":13,"area":null},{"id":"2310.12462","title":"Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights","created":"2023-10-19","cited_by_count":12,"area":null},{"id":"2412.18040","title":"Theoretical Constraints on the Expressive Power of $\\mathsf{RoPE}$-based Tensor Attention Transformers","created":"2024-12-23","cited_by_count":12,"area":null},{"id":"2501.19215","title":"Strassen Attention, Split VC Dimension and Compositionality in Transformers","created":"2025-01-31","cited_by_count":12,"area":null},{"id":"2405.13718","title":"Next-token prediction capacity: general upper bounds and a lower bound for transformers","created":"2024-05-22","cited_by_count":11,"area":null},{"id":"2503.01805","title":"Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers","created":"2025-03-03","cited_by_count":11,"area":null},{"id":"2409.17677","title":"On the Optimal Memorization Capacity of Transformers","created":"2024-09-26","cited_by_count":10,"area":null},{"id":"2503.09069","title":"Theoretical Guarantees for High Order Trajectory Refinement in Generative Flows","created":"2025-03-12","cited_by_count":10,"area":"diffusion"},{"id":"2602.03655","title":"Sequential Group Composition: A Window into the Mechanics of Deep Learning","created":"2026-02-03","cited_by_count":10,"area":null},{"id":"2311.15390","title":"Local Convergence of Approximate Newton Method for Two Layer Nonlinear Regression","created":"2023-11-26","cited_by_count":9,"area":null},{"id":"2410.04271","title":"Fundamental Limitations on Subquadratic Alternatives to Transformers","created":"2024-10-05","cited_by_count":9,"area":"ssm"},{"id":"2410.09605","title":"Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis","created":"2024-10-12","cited_by_count":9,"area":null},{"id":"2411.02344","title":"Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning","created":"2024-11-04","cited_by_count":9,"area":"lr"},{"id":"2505.08140","title":"Lost in Transmission: When and Why LLMs Fail to Reason Globally","created":"2025-05-13","cited_by_count":9,"area":"lr"},{"id":"2506.02651","title":"Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks","created":"2025-06-03","cited_by_count":9,"area":null},{"id":"2404.02040","title":"Transformers as Transducers","created":"2024-04-02","cited_by_count":8,"area":null},{"id":"2404.03380","title":"On the Theoretical Expressive Power and the Design Space of Higher-Order Graph Transformers","created":"2024-04-04","cited_by_count":8,"area":null},{"id":"2502.16792","title":"The Role of Sparsity for Length Generalization in Transformers","created":"2025-02-24","cited_by_count":8,"area":null},{"id":"2505.21024","title":"Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers","created":"2025-05-27","cited_by_count":8,"area":null},{"id":"2507.02754","title":"Fast and Simplex: 2-Simplicial Attention in Triton","created":"2025-07-03","cited_by_count":8,"area":null},{"id":"2605.05971","title":"Training Transformers for KV Cache Compressibility","created":"2026-05-07","cited_by_count":7,"area":"inference"},{"id":"2308.08358","title":"Convergence of Two-Layer Regression with Nonlinear Units","created":"2023-08-16","cited_by_count":6,"area":null},{"id":"2311.13624","title":"A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer","created":"2023-11-22","cited_by_count":6,"area":null},{"id":"2311.14652","title":"One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space","created":"2023-11-24","cited_by_count":6,"area":null},{"id":"2408.15185","title":"Human-Centric Video Anomaly Detection Through Spatio-Temporal Pose Tokenization and Transformer","created":"2024-08-27","cited_by_count":6,"area":null},{"id":"2503.01544","title":"Compositional Reasoning with Transformers, RNNs, and Chain of Thought","created":"2025-03-03","cited_by_count":6,"area":"lr"},{"id":"2505.21785","title":"Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities","created":"2025-05-27","cited_by_count":6,"area":null},{"id":"2506.05200","title":"Transformers Meet In-Context Learning: A Universal Approximation Theory","created":"2025-06-05","cited_by_count":6,"area":null},{"id":"2510.00368","title":"The Transformer Cookbook","created":"2025-10-01","cited_by_count":6,"area":null},{"id":"2511.11579","title":"Decoupling Positional and Symbolic Attention Behavior in Transformers","created":"2025-10-03","cited_by_count":6,"area":null},{"id":"2407.16153","title":"On the Benefits of Rank in Attention Layers","created":"2024-07-23","cited_by_count":5,"area":null},{"id":"2410.01600","title":"ENTP: Encoder-only Next Token Prediction","created":"2024-10-02","cited_by_count":5,"area":null},{"id":"2502.15955","title":"Compression Barriers for Autoregressive Transformers","created":"2025-02-21","cited_by_count":5,"area":"inference"},{"id":"2504.13558","title":"Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective","created":"2025-04-18","cited_by_count":5,"area":null},{"id":"2505.14840","title":"Subquadratic Algorithms and Hardness for Attention with Any Temperature","created":"2025-05-20","cited_by_count":5,"area":null},{"id":"2505.20802","title":"Leaner Transformers: More Heads, Less Depth","created":"2025-05-27","cited_by_count":5,"area":null},{"id":"2507.07544","title":"Position: We Need An Algorithmic Understanding of Generative AI","created":"2025-07-10","cited_by_count":5,"area":null},{"id":"2602.04768","title":"Billion-Scale Graph Foundation Models","created":"2026-02-04","cited_by_count":5,"area":null},{"id":"2602.05896","title":"Parity, Sensitivity, and Transformers","created":"2026-02-05","cited_by_count":5,"area":null},{"id":"2412.01417","title":"Learning Elementary Cellular Automata with Transformers","created":"2024-12-02","cited_by_count":4,"area":null},{"id":"2501.10688","title":"Neural Algorithmic Reasoning for Hypergraphs with Looped Transformers","created":"2025-01-18","cited_by_count":4,"area":"lr"},{"id":"2508.16745","title":"Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling","created":"2025-08-22","cited_by_count":4,"area":"lr"},{"id":"2510.05364","title":"The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures","created":"2025-10-06","cited_by_count":4,"area":null},{"id":"2510.18870","title":"Triangle Multiplication Is All You Need For Biomolecular Structure Representations","created":"2025-10-21","cited_by_count":4,"area":null},{"id":"2407.03250","title":"When big data actually are low-rank, or entrywise approximation of certain function-generated matrices","created":"2024-07-03","cited_by_count":3,"area":null},{"id":"2409.11232","title":"Fast Analysis of the OpenAI O1-Preview Model in Solving Random K-SAT Problem: Does the LLM Solve the Problem Itself or Call an External SAT Solver?","created":"2024-09-17","cited_by_count":3,"area":null},{"id":"2409.12446","title":"Neural Networks Generalize on Low Complexity Data","created":"2024-09-19","cited_by_count":3,"area":null},{"id":"2409.18028","title":"Compositional Hardness of Code in Large Language Models -- A Probabilistic Perspective","created":"2024-09-26","cited_by_count":3,"area":null},{"id":"2411.13028","title":"A Theory for Compressibility of Graph Transformers for Transductive Learning","created":"2024-11-20","cited_by_count":3,"area":null},{"id":"2412.20195","title":"Lower bounds on transformers with infinite precision","created":"2024-12-28","cited_by_count":3,"area":null},{"id":"2501.02535","title":"A completely uniform transformer for parity","created":"2025-01-05","cited_by_count":3,"area":null},{"id":"2502.07553","title":"Transformers Provably Learn Sparse XOR with Polylogarithmic Parameters","created":"2025-02-11","cited_by_count":3,"area":null},{"id":"2503.10542","title":"Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More","created":"2025-03-13","cited_by_count":3,"area":null},{"id":"2503.11272","title":"When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective","created":"2025-03-14","cited_by_count":3,"area":null},{"id":"2503.22076","title":"Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)","created":"2025-03-28","cited_by_count":3,"area":null},{"id":"2509.09001","title":"Fast attention mechanisms: a tale of parallelism","created":"2025-09-10","cited_by_count":3,"area":null},{"id":"2510.08389","title":"Revisiting Hallucination Detection with Effective Rank-based Uncertainty","created":"2025-10-09","cited_by_count":3,"area":null},{"id":"2601.17958","title":"TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors","created":"2026-01-25","cited_by_count":3,"area":null},{"id":"2602.11175","title":"Barriers to Discrete Reasoning with Transformers: A Survey Across Depth, Exactness, and Bandwidth","created":"2026-01-19","cited_by_count":3,"area":null},{"id":"2604.17935","title":"How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers","created":"2026-04-20","cited_by_count":3,"area":null},{"id":"2503.10571","title":"Radar: Fast Long-Context Decoding for Any Transformer","created":"2025-03-13","cited_by_count":2,"area":null},{"id":"2503.11108","title":"Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding","created":"2025-03-14","cited_by_count":2,"area":"inference"}],"next_cursor":"100"}