{"total":196,"rows":[{"id":"2102.12092","title":"Zero-Shot Text-to-Image Generation","created":"2021-02-24","cited_by_count":4288,"area":null},{"id":"2106.11959","title":"Revisiting Deep Learning Models for Tabular Data","created":"2021-06-22","cited_by_count":740,"area":null},{"id":"2104.10972","title":"ImageNet-21K Pretraining for the Masses","created":"2021-04-22","cited_by_count":613,"area":null},{"id":"2110.13985","title":"Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers","created":"2021-10-26","cited_by_count":608,"area":null},{"id":"2106.04554","title":"A Survey of Transformers","created":"2021-06-08","cited_by_count":401,"area":null},{"id":"2403.03853","title":"ShortGPT: Layers in Large Language Models are More Redundant Than You Expect","created":"2024-03-06","cited_by_count":383,"area":null},{"id":"2305.17333","title":"Fine-Tuning Language Models with Just Forward Passes","created":"2023-05-27","cited_by_count":345,"area":null},{"id":"2202.07925","title":"ActionFormer: Localizing Moments of Actions with Transformers","created":"2022-02-16","cited_by_count":311,"area":null},{"id":"2203.03466","title":"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer","created":"2022-03-07","cited_by_count":288,"area":null},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265,"area":null},{"id":"2111.05464","title":"Are Transformers More Robust Than CNNs?","created":"2021-11-10","cited_by_count":187,"area":null},{"id":"2203.00555","title":"DeepNet: Scaling Transformers to 1,000 Layers","created":"2022-03-01","cited_by_count":178,"area":null},{"id":"2303.06296","title":"Stabilizing Transformer Training by Preventing Attention Entropy Collapse","created":"2023-03-11","cited_by_count":161,"area":null},{"id":"2203.06717","title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","created":"2022-03-13","cited_by_count":155,"area":null},{"id":"2206.03126","title":"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse","created":"2022-06-07","cited_by_count":141,"area":null},{"id":"2210.13452","title":"MetaFormer Baselines for Vision","created":"2022-10-24","cited_by_count":132,"area":null},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124,"area":null},{"id":"2210.05643","title":"A Kernel-Based View of Language Model Fine-Tuning","created":"2022-10-11","cited_by_count":121,"area":null},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114,"area":null},{"id":"2304.03208","title":"Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster","created":"2023-04-06","cited_by_count":109,"area":null},{"id":"2207.14484","title":"Adaptive Gradient Methods at the Edge of Stability","created":"2022-07-29","cited_by_count":101,"area":null},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98,"area":null},{"id":"2112.01838","title":"Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer","created":"2021-12-03","cited_by_count":91,"area":null},{"id":"2108.02388","title":"TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding","created":"2021-08-05","cited_by_count":86,"area":null},{"id":"2303.04245","title":"How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding","created":"2023-03-07","cited_by_count":86,"area":null},{"id":"2104.06022","title":"Lessons on Parameter Sharing across Layers in Transformers","created":"2021-04-13","cited_by_count":85,"area":null},{"id":"2409.19606","title":"Hyper-Connections","created":"2024-09-29","cited_by_count":84,"area":null},{"id":"2110.09456","title":"NormFormer: Improved Transformer Pretraining with Extra Normalization","created":"2021-10-18","cited_by_count":83,"area":null},{"id":"2212.14034","title":"Cramming: Training a Language Model on a Single GPU in One Day","created":"2022-12-28","cited_by_count":81,"area":null},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80,"area":null},{"id":"2103.14636","title":"A Practical Survey on Faster and Lighter Transformers","created":"2021-03-26","cited_by_count":73,"area":null},{"id":"2106.14342","title":"Stabilizing Equilibrium Models by Jacobian Regularization","created":"2021-06-28","cited_by_count":72,"area":null},{"id":"2306.13089","title":"GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning","created":"2023-05-28","cited_by_count":70,"area":null},{"id":"2409.03752","title":"Attention Heads of Large Language Models: A Survey","created":"2024-09-05","cited_by_count":67,"area":null},{"id":"2111.00035","title":"Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\\\"om Method","created":"2021-10-29","cited_by_count":63,"area":null},{"id":"2502.05795","title":"The Curse of Depth in Large Language Models","created":"2025-02-09","cited_by_count":62,"area":null},{"id":"2106.10207","title":"Distributed Deep Learning in Open Collaborations","created":"2021-06-18","cited_by_count":58,"area":null},{"id":"2312.16903","title":"Spike No More: Stabilizing the Pre-training of Large Language Models","created":"2023-12-28","cited_by_count":52,"area":null},{"id":"2105.09447","title":"VTNet: Visual Transformer Network for Object Goal Navigation","created":"2021-05-20","cited_by_count":51,"area":null},{"id":"2402.11867","title":"LoRA Training in the NTK Regime has No Spurious Local Minima","created":"2024-02-19","cited_by_count":48,"area":null},{"id":"2109.12218","title":"Long-Range Transformers for Dynamic Spatiotemporal Forecasting","created":"2021-09-24","cited_by_count":46,"area":null},{"id":"2205.01138","title":"Transformers in Time-series Analysis: A Tutorial","created":"2022-04-28","cited_by_count":46,"area":null},{"id":"2302.10322","title":"Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation","created":"2023-02-20","cited_by_count":45,"area":null},{"id":"2102.08098","title":"GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training","created":"2021-02-16","cited_by_count":44,"area":null},{"id":"2402.10198","title":"SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention","created":"2024-02-15","cited_by_count":44,"area":null},{"id":"2110.04369","title":"A Loss Curvature Perspective on Training Instability in Deep Learning","created":"2021-10-08","cited_by_count":40,"area":null},{"id":"2303.09859","title":"Trained on 100 million words and still in shape: BERT meets British National Corpus","created":"2023-03-17","cited_by_count":39,"area":null},{"id":"2303.11607","title":"Transformers in Speech Processing: A Survey","created":"2023-03-21","cited_by_count":39,"area":null},{"id":"2309.03696","title":"Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory","created":"2023-09-07","cited_by_count":38,"area":null},{"id":"2105.14528","title":"Fast Nearest Neighbor Machine Translation","created":"2021-05-30","cited_by_count":37,"area":null},{"id":"2212.08136","title":"Efficient Long Sequence Modeling via State Space Augmented Transformer","created":"2022-12-15","cited_by_count":37,"area":null},{"id":"2311.01906","title":"Simplifying Transformer Blocks","created":"2023-11-03","cited_by_count":37,"area":null},{"id":"2412.13795","title":"Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN","created":"2024-12-18","cited_by_count":37,"area":null},{"id":"2102.12459","title":"When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute","created":"2021-02-24","cited_by_count":34,"area":null},{"id":"2304.09856","title":"LipsFormer: Introducing Lipschitz Continuity to Vision Transformers","created":"2023-04-19","cited_by_count":34,"area":null},{"id":"2304.14802","title":"ResiDual: Transformer with Dual Residual Connections","created":"2023-04-28","cited_by_count":31,"area":null},{"id":"2502.12170","title":"MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections","created":"2025-02-13","cited_by_count":31,"area":null},{"id":"2305.14858","title":"Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers","created":"2023-05-24","cited_by_count":30,"area":null},{"id":"2312.01429","title":"Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars","created":"2023-12-03","cited_by_count":30,"area":null},{"id":"2202.00980","title":"Robust Training of Neural Networks Using Scale Invariant Architectures","created":"2022-02-02","cited_by_count":29,"area":null},{"id":"2203.09176","title":"ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation","created":"2022-03-17","cited_by_count":27,"area":null},{"id":"2301.08125","title":"Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification","created":"2023-01-19","cited_by_count":27,"area":null},{"id":"2306.06722","title":"$E(2)$-Equivariant Vision Transformer","created":"2023-06-11","cited_by_count":27,"area":null},{"id":"2302.01172","title":"STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition","created":"2023-02-02","cited_by_count":26,"area":null},{"id":"2106.05956","title":"Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning","created":"2021-06-10","cited_by_count":25,"area":null},{"id":"2106.12102","title":"LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction","created":"2021-06-23","cited_by_count":25,"area":null},{"id":"2303.08046","title":"Ultra-High-Resolution Detector Simulation with Intra-Event Aware GAN and Self-Supervised Relational Reasoning","created":"2023-03-07","cited_by_count":25,"area":null},{"id":"2505.19227","title":"Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law","created":"2025-05-25","cited_by_count":24,"area":null},{"id":"2305.12475","title":"Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods","created":"2023-05-21","cited_by_count":23,"area":null},{"id":"2105.07205","title":"Rethinking Skip Connection with Layer Normalization in Transformers and ResNets","created":"2021-05-15","cited_by_count":22,"area":null},{"id":"2107.08031","title":"Is attention to bounding boxes all you need for pedestrian action prediction?","created":"2021-07-16","cited_by_count":21,"area":null},{"id":"2204.06644","title":"METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals","created":"2022-04-13","cited_by_count":20,"area":null},{"id":"2202.02842","title":"Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data","created":"2022-02-06","cited_by_count":19,"area":null},{"id":"2210.06423","title":"Foundation Transformers","created":"2022-10-12","cited_by_count":19,"area":null},{"id":"2405.05409","title":"Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing","created":"2024-05-08","cited_by_count":17,"area":null},{"id":"2106.09650","title":"Multi-head or Single-head? An Empirical Comparison for Transformer Training","created":"2021-06-17","cited_by_count":16,"area":null},{"id":"2206.00330","title":"B2T Connection: Serving Stability and Performance in Deep Transformers","created":"2022-06-01","cited_by_count":16,"area":null},{"id":"2503.04598","title":"HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization","created":"2025-03-06","cited_by_count":16,"area":null},{"id":"2303.00579","title":"Are More Layers Beneficial to Graph Transformers?","created":"2023-03-01","cited_by_count":15,"area":null},{"id":"2211.03808","title":"ToDD: Topological Compound Fingerprinting in Computer-Aided Drug Discovery","created":"2022-11-07","cited_by_count":13,"area":null},{"id":"2504.07866","title":"Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs","created":"2025-04-10","cited_by_count":13,"area":null},{"id":"2104.02308","title":"ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation","created":"2021-04-06","cited_by_count":12,"area":null},{"id":"2303.14926","title":"Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation","created":"2023-03-27","cited_by_count":12,"area":null},{"id":"2307.05979","title":"Transformers in Reinforcement Learning: A Survey","created":"2023-07-12","cited_by_count":12,"area":null},{"id":"2411.07538","title":"Unraveling the Gradient Descent Dynamics of Transformers","created":"2024-11-12","cited_by_count":11,"area":null},{"id":"2109.06679","title":"Efficient Inference for Multilingual Neural Machine Translation","created":"2021-09-14","cited_by_count":10,"area":null},{"id":"2207.14467","title":"GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation","created":"2022-07-29","cited_by_count":10,"area":null},{"id":"2405.02803","title":"Is Flash Attention Stable?","created":"2024-05-05","cited_by_count":10,"area":null},{"id":"2602.08064","title":"SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm","created":"2026-02-08","cited_by_count":10,"area":null},{"id":"2302.01128","title":"Mnemosyne: Learning to Train Transformers with Transformers","created":"2023-02-02","cited_by_count":9,"area":null},{"id":"2305.15348","title":"READ: Recurrent Adaptation of Large Transformers","created":"2023-05-24","cited_by_count":9,"area":null},{"id":"2307.09361","title":"MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments","created":"2023-07-18","cited_by_count":9,"area":null},{"id":"2410.10986","title":"What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis","created":"2024-10-14","cited_by_count":9,"area":null},{"id":"2411.07176","title":"More Expressive Attention with Negative Weights","created":"2024-11-11","cited_by_count":9,"area":null},{"id":"2505.21910","title":"Taming Transformer Without Using Learning Rate Warmup","created":"2025-05-28","cited_by_count":9,"area":null},{"id":"2605.20613","title":"HRM-Text: Efficient Pretraining Beyond Scaling","created":"2026-05-20","cited_by_count":9,"area":null},{"id":"2109.02797","title":"Puzzle Solving without Search or Human Knowledge: An Unnatural Language Approach","created":"2021-09-07","cited_by_count":8,"area":null},{"id":"2112.08152","title":"Faster Nearest Neighbor Machine Translation","created":"2021-12-15","cited_by_count":8,"area":null},{"id":"2204.03475","title":"Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results","created":"2022-04-07","cited_by_count":8,"area":null},{"id":"2304.08968","title":"Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMs","created":"2023-04-18","cited_by_count":8,"area":null}],"next_cursor":"100"}