{"total":141,"rows":[{"id":"2303.06296","title":"Stabilizing Transformer Training by Preventing Attention Entropy Collapse","created":"2023-03-11","cited_by_count":161,"area":null},{"id":"2504.02732","title":"Why do LLMs attend to the first token?","created":"2025-04-03","cited_by_count":127,"area":null},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124,"area":null},{"id":"2305.19420","title":"What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization","created":"2023-05-30","cited_by_count":118,"area":null},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80,"area":null},{"id":"2505.01618","title":"Don't be lazy: CompleteP enables compute-efficient deep transformers","created":"2025-05-02","cited_by_count":78,"area":null},{"id":"2305.15805","title":"Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers","created":"2023-05-25","cited_by_count":67,"area":null},{"id":"2309.16620","title":"Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit","created":"2023-09-28","cited_by_count":66,"area":null},{"id":"2502.05795","title":"The Curse of Depth in Large Language Models","created":"2025-02-09","cited_by_count":62,"area":null},{"id":"2405.18781","title":"On the Role of Attention Masks and LayerNorm in Transformers","created":"2024-05-29","cited_by_count":48,"area":null},{"id":"2306.17759","title":"The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit","created":"2023-06-30","cited_by_count":47,"area":null},{"id":"2302.10322","title":"Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation","created":"2023-02-20","cited_by_count":45,"area":null},{"id":"2402.10198","title":"SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention","created":"2024-02-15","cited_by_count":44,"area":null},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40,"area":null},{"id":"2405.15712","title":"Infinite Limits of Multi-head Transformer Dynamics","created":"2024-05-24","cited_by_count":39,"area":null},{"id":"2311.01906","title":"Simplifying Transformer Blocks","created":"2023-11-03","cited_by_count":37,"area":null},{"id":"2512.01868","title":"The Mean-Field Dynamics of Transformers","created":"2025-12-01","cited_by_count":30,"area":null},{"id":"2411.04551","title":"Measure-to-measure interpolation using Transformers","created":"2024-11-07","cited_by_count":29,"area":null},{"id":"2606.18206","title":"Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers","created":"2026-06-16","cited_by_count":28,"area":null},{"id":"2308.12247","title":"How to Protect Copyright Data in Optimization of Large Language Models?","created":"2023-08-23","cited_by_count":26,"area":null},{"id":"2509.25040","title":"A multiscale analysis of mean-field transformers in the moderate interaction regime","created":"2025-09-29","cited_by_count":25,"area":null},{"id":"2403.02579","title":"Geometric Dynamics of Signal Propagation Predict Trainability of Transformers","created":"2024-03-05","cited_by_count":24,"area":null},{"id":"2310.07999","title":"LEMON: Lossless model expansion","created":"2023-10-12","cited_by_count":23,"area":"lr"},{"id":"2402.17457","title":"Super Consistency of Neural Network Landscapes and Learning Rate Transfer","created":"2024-02-27","cited_by_count":23,"area":null},{"id":"2401.04301","title":"Setting the Record Straight on Transformer Oversmoothing","created":"2024-01-09","cited_by_count":21,"area":null},{"id":"2506.16640","title":"Long-Context Generalization with Sparse Attention","created":"2025-06-19","cited_by_count":21,"area":null},{"id":"2510.05554","title":"Critical attention scaling in long-context transformers","created":"2025-10-07","cited_by_count":20,"area":null},{"id":"2402.02098","title":"Self-attention Networks Localize When QK-eigenspectrum Concentrates","created":"2024-02-03","cited_by_count":19,"area":null},{"id":"2308.16800","title":"Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks","created":"2023-08-31","cited_by_count":18,"area":null},{"id":"2505.24333","title":"Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation","created":"2025-05-30","cited_by_count":17,"area":null},{"id":"2502.00919","title":"Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings","created":"2025-02-02","cited_by_count":16,"area":null},{"id":"2503.04598","title":"HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization","created":"2025-03-06","cited_by_count":16,"area":null},{"id":"2503.05657","title":"NoT: Federated Unlearning via Weight Negation","created":"2025-03-07","cited_by_count":15,"area":"safety"},{"id":"2506.12543","title":"Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling","created":"2025-06-14","cited_by_count":15,"area":null},{"id":"2310.12956","title":"Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems","created":"2023-10-19","cited_by_count":14,"area":null},{"id":"2305.18399","title":"On the impact of activation and normalization in obtaining isometric embeddings at initialization","created":"2023-05-28","cited_by_count":13,"area":null},{"id":"2402.05173","title":"Towards Understanding Inductive Bias in Transformers: A View From Infinity","created":"2024-02-07","cited_by_count":13,"area":null},{"id":"2604.09780","title":"The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise","created":"2026-04-10","cited_by_count":13,"area":"moe"},{"id":"2312.01648","title":"Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation","created":"2023-12-04","cited_by_count":12,"area":null},{"id":"2404.08634","title":"When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models","created":"2024-04-12","cited_by_count":12,"area":null},{"id":"2502.20126","title":"FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute","created":"2025-02-27","cited_by_count":12,"area":"diffusion"},{"id":"2506.01115","title":"Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer","created":"2025-06-01","cited_by_count":12,"area":"lr"},{"id":"2312.04234","title":"Graph Convolutions Enrich the Self-Attention in Transformers!","created":"2023-12-07","cited_by_count":11,"area":null},{"id":"2411.07538","title":"Unraveling the Gradient Descent Dynamics of Transformers","created":"2024-11-12","cited_by_count":11,"area":null},{"id":"2507.07814","title":"Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers","created":"2025-07-10","cited_by_count":11,"area":null},{"id":"2405.10616","title":"Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization","created":"2024-05-17","cited_by_count":10,"area":null},{"id":"2405.19279","title":"Understanding and Minimising Outlier Features in Neural Network Training","created":"2024-05-29","cited_by_count":10,"area":null},{"id":"2407.04358","title":"An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes","created":"2024-07-05","cited_by_count":10,"area":null},{"id":"2410.02247","title":"Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization","created":"2024-10-03","cited_by_count":9,"area":null},{"id":"2410.10986","title":"What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis","created":"2024-10-14","cited_by_count":9,"area":null},{"id":"2505.21910","title":"Taming Transformer Without Using Learning Rate Warmup","created":"2025-05-28","cited_by_count":9,"area":null},{"id":"2510.22026","title":"Normalization in Attention Dynamics","created":"2025-10-24","cited_by_count":9,"area":null},{"id":"2310.02012","title":"Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion","created":"2023-10-03","cited_by_count":8,"area":null},{"id":"2403.06871","title":"On the Generalization Ability of Unsupervised Pretraining","created":"2024-03-11","cited_by_count":8,"area":null},{"id":"2510.00345","title":"Cutting the Skip: Training Residual-Free Transformers","created":"2025-09-30","cited_by_count":8,"area":null},{"id":"2511.11553","title":"Multistability of Self-Attention Dynamics in Transformers","created":"2025-11-14","cited_by_count":8,"area":null},{"id":"2512.14366","title":"Optimizing Rank for High-Fidelity Implicit Neural Representations","created":"2025-12-16","cited_by_count":8,"area":null},{"id":"2601.21942","title":"Clustering in Deep Stochastic Transformers","created":"2026-01-29","cited_by_count":8,"area":null},{"id":"2402.15415","title":"Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics","created":"2024-02-23","cited_by_count":7,"area":null},{"id":"2505.16284","title":"Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse","created":"2025-05-22","cited_by_count":7,"area":null},{"id":"2506.13688","title":"What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers","created":"2025-06-16","cited_by_count":7,"area":null},{"id":"2506.24106","title":"On the Predictive Power of Representation Dispersion in Language Models","created":"2025-06-30","cited_by_count":7,"area":null},{"id":"2504.16275","title":"Quantum Doubly Stochastic Transformers","created":"2025-04-22","cited_by_count":6,"area":null},{"id":"2505.19985","title":"Structured Initialization for Vision Transformers","created":"2025-05-26","cited_by_count":6,"area":null},{"id":"2506.05249","title":"On the Convergence of Gradient Descent on Learning Transformers with Residual Connections","created":"2025-06-05","cited_by_count":6,"area":null},{"id":"2602.04852","title":"On State Reduction in Linear Attention","created":"2026-02-04","cited_by_count":6,"area":null},{"id":"2505.19458","title":"Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians","created":"2025-05-26","cited_by_count":5,"area":null},{"id":"2602.01605","title":"Universal Redundancies in Time Series Foundation Models","created":"2026-02-02","cited_by_count":5,"area":null},{"id":"2602.05929","title":"KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs","created":"2026-02-05","cited_by_count":5,"area":"inference"},{"id":"2608.08888","title":"Full-bandwidth transformer","created":"2026-08-09","cited_by_count":5,"area":null},{"id":"2311.03233","title":"Navigating Scaling Laws: Compute Optimality in Adaptive Model Training","created":"2023-11-06","cited_by_count":4,"area":null},{"id":"2410.05052","title":"Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes","created":"2024-10-07","cited_by_count":4,"area":null},{"id":"2410.10609","title":"Lambda-Skip Connections: the architectural component that prevents Rank Collapse","created":"2024-10-14","cited_by_count":4,"area":null},{"id":"2411.05746","title":"Continuous-Time Analysis of Adaptive Optimization and Normalization","created":"2024-11-08","cited_by_count":4,"area":null},{"id":"2402.02634","title":"Key-Graph Transformer for Image Restoration","created":"2024-02-04","cited_by_count":3,"area":null},{"id":"2408.13152","title":"Long-term Pre-training for Temporal Action Detection with Transformers","created":"2024-08-23","cited_by_count":3,"area":null},{"id":"2410.08024","title":"Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling","created":"2024-10-10","cited_by_count":3,"area":null},{"id":"2410.13060","title":"AERO: Entropy-Guided Framework for Private LLM Inference","created":"2024-10-16","cited_by_count":3,"area":null},{"id":"2503.22547","title":"Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation","created":"2025-03-28","cited_by_count":3,"area":null},{"id":"2508.18387","title":"Integral Transformer: Denoising Attention, Not Too Much Not Too Little","created":"2025-08-25","cited_by_count":3,"area":null},{"id":"2511.21477","title":"Frequency-Aware Token Reduction for Efficient Vision Transformer","created":"2025-11-26","cited_by_count":3,"area":"inference"},{"id":"2602.18849","title":"Exact Attention Sensitivity and the Geometry of Transformer Stability","created":"2026-02-21","cited_by_count":3,"area":null},{"id":"2602.20433","title":"Disentangling Geometry, Performance, and Training in Language Models","created":"2026-02-24","cited_by_count":3,"area":null},{"id":"2603.06922","title":"NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks","created":"2026-03-06","cited_by_count":3,"area":null},{"id":"2606.19491","title":"Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale","created":"2026-06-17","cited_by_count":3,"area":null},{"id":"2410.15578","title":"Generalized Probabilistic Attention Mechanism in Transformers","created":"2024-10-21","cited_by_count":2,"area":null},{"id":"2411.01443","title":"Activating Self-Attention for Multi-Scene Absolute Pose Regression","created":"2024-11-03","cited_by_count":2,"area":null},{"id":"2503.18337","title":"Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models","created":"2025-03-24","cited_by_count":2,"area":null},{"id":"2508.00127","title":"Structured Transformations for Stable and Interpretable Neural Computation","created":"2025-07-31","cited_by_count":2,"area":null},{"id":"2601.04670","title":"Learning Dynamics in RL Post-Training for Language Models","created":"2026-01-08","cited_by_count":2,"area":"lr"},{"id":"2601.20769","title":"Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence","created":"2026-01-28","cited_by_count":2,"area":null},{"id":"2601.20994","title":"The Depth Delusion: Why Transformers Should Be Wider, Not Deeper","created":"2026-01-28","cited_by_count":2,"area":null},{"id":"2604.01609","title":"Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression","created":"2026-04-02","cited_by_count":2,"area":"inference"},{"id":"2604.07925","title":"Sinkhorn doubly stochastic attention rank decay analysis","created":"2026-04-09","cited_by_count":2,"area":null},{"id":"2605.27662","title":"How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks","created":"2026-05-26","cited_by_count":2,"area":null},{"id":"2606.21158","title":"Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity","created":"2026-06-19","cited_by_count":2,"area":null},{"id":"2607.14018","title":"Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth","created":"2026-07-15","cited_by_count":2,"area":null},{"id":"2312.06182","title":"Why \"classic\" Transformers are shallow and how to make them go deep","created":"2023-12-11","cited_by_count":1,"area":null},{"id":"2410.20107","title":"Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activations","created":"2024-10-26","cited_by_count":1,"area":null},{"id":"2504.18929","title":"Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity","created":"2025-04-26","cited_by_count":1,"area":null}],"next_cursor":"100"}