{"total":119,"rows":[{"id":"2102.03902","title":"Nystr\\\"omformer: A Nystr\\\"om-Based Algorithm for Approximating Self-Attention","created":"2021-02-07","cited_by_count":471,"area":null},{"id":"2103.02143","title":"Random Feature Attention","created":"2021-03-03","cited_by_count":354,"area":null},{"id":"2209.10655","title":"Mega: Moving Average Equipped Gated Attention","created":"2022-09-21","cited_by_count":179,"area":null},{"id":"2103.05247","title":"Pretrained Transformers as Universal Computation Engines","created":"2021-03-09","cited_by_count":162,"area":null},{"id":"2110.15343","title":"Scatterbrain: Unifying Sparse and Low-rank Attention Approximation","created":"2021-10-28","cited_by_count":130,"area":null},{"id":"2110.11945","title":"SOFT: Softmax-free Transformer with Linear Complexity","created":"2021-10-22","cited_by_count":128,"area":null},{"id":"2107.02192","title":"Long-Short Transformer: Efficient Transformers for Language and Vision","created":"2021-07-05","cited_by_count":102,"area":null},{"id":"2106.01540","title":"Luna: Linear Unified Nested Attention","created":"2021-06-03","cited_by_count":96,"area":null},{"id":"2106.06295","title":"Going Beyond Linear Transformers with Recurrent Fast Weight Programmers","created":"2021-06-11","cited_by_count":89,"area":null},{"id":"2103.14636","title":"A Practical Survey on Faster and Lighter Transformers","created":"2021-03-26","cited_by_count":73,"area":null},{"id":"2202.06258","title":"Flowformer: Linearizing Transformers with Conservation Flows","created":"2022-02-13","cited_by_count":69,"area":null},{"id":"2111.00035","title":"Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\\\"om Method","created":"2021-10-29","cited_by_count":63,"area":null},{"id":"2110.07732","title":"The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization","created":"2021-10-14","cited_by_count":62,"area":null},{"id":"2106.06899","title":"Memory-efficient Transformers via Top-$k$ Attention","created":"2021-06-13","cited_by_count":60,"area":null},{"id":"2404.08801","title":"Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length","created":"2024-04-12","cited_by_count":46,"area":null},{"id":"2105.08399","title":"Relative Positional Encoding for Transformers with Linear Complexity","created":"2021-05-18","cited_by_count":37,"area":null},{"id":"2212.08136","title":"Efficient Long Sequence Modeling via State Space Augmented Transformer","created":"2022-12-15","cited_by_count":37,"area":null},{"id":"2406.06467","title":"How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad","created":"2024-06-10","cited_by_count":36,"area":null},{"id":"2310.02980","title":"Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors","created":"2023-10-04","cited_by_count":34,"area":null},{"id":"2501.15061","title":"PolaFormer: Polarity-aware Linear Attention for Vision Transformers","created":"2025-01-25","cited_by_count":32,"area":null},{"id":"2109.15142","title":"Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems","created":"2021-09-30","cited_by_count":31,"area":null},{"id":"2204.04667","title":"Linear Complexity Randomized Self-attention Mechanism","created":"2022-04-10","cited_by_count":31,"area":null},{"id":"2304.08612","title":"Bridging Discrete and Backpropagation: Straight-Through and Beyond","created":"2023-04-17","cited_by_count":31,"area":null},{"id":"2305.19798","title":"Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation","created":"2023-05-31","cited_by_count":29,"area":null},{"id":"2110.08678","title":"Improving Transformers with Probabilistic Attention Keys","created":"2021-10-16","cited_by_count":28,"area":null},{"id":"2108.02347","title":"FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention","created":"2021-08-05","cited_by_count":26,"area":null},{"id":"2305.15947","title":"Online learning of long-range dependencies","created":"2023-05-25","cited_by_count":25,"area":null},{"id":"2306.11197","title":"Sparse Modular Activation for Efficient Sequence Modeling","created":"2023-06-19","cited_by_count":20,"area":null},{"id":"2206.00826","title":"BayesFormer: Transformer with Uncertainty Estimation","created":"2022-06-02","cited_by_count":19,"area":null},{"id":"2401.00955","title":"Learning Long Sequences in Spiking Neural Networks","created":"2023-12-14","cited_by_count":19,"area":null},{"id":"2405.06147","title":"State-Free Inference of State-Space Models: The Transfer Function Approach","created":"2024-05-10","cited_by_count":18,"area":null},{"id":"2111.09714","title":"You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling","created":"2021-11-18","cited_by_count":17,"area":null},{"id":"2109.01982","title":"Learning Hierarchical Structures with Differentiable Nondeterministic Stacks","created":"2021-09-05","cited_by_count":16,"area":null},{"id":"2106.06038","title":"Modeling Hierarchical Structures with Continuous Recursive Neural Networks","created":"2021-06-10","cited_by_count":15,"area":null},{"id":"2402.12138","title":"Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers","created":"2024-02-19","cited_by_count":11,"area":null},{"id":"2404.00560","title":"A Theory for Length Generalization in Learning to Reason","created":"2024-03-31","cited_by_count":11,"area":null},{"id":"2405.02670","title":"From Generalization Analysis to Optimization Designs for State Space Models","created":"2024-05-04","cited_by_count":11,"area":null},{"id":"2112.02624","title":"Dynamic Token Normalization Improves Vision Transformers","created":"2021-12-05","cited_by_count":10,"area":null},{"id":"2204.05424","title":"A Call for Clarity in Beam Search: How It Works and When It Stops","created":"2022-04-11","cited_by_count":10,"area":null},{"id":"2310.11984","title":"From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers","created":"2023-10-18","cited_by_count":10,"area":null},{"id":"2411.02824","title":"Layer-Adaptive State Pruning for Deep State Space Models","created":"2024-11-05","cited_by_count":10,"area":null},{"id":"2412.17810","title":"Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction","created":"2024-12-23","cited_by_count":10,"area":null},{"id":"2207.03341","title":"Softmax-free Linear Transformers","created":"2022-07-05","cited_by_count":9,"area":null},{"id":"2402.17396","title":"Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies","created":"2024-02-27","cited_by_count":9,"area":null},{"id":"2206.07235","title":"Training Discrete Deep Generative Models via Gapped Straight-Through Estimator","created":"2022-06-15","cited_by_count":8,"area":null},{"id":"2405.02842","title":"IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs","created":"2024-05-05","cited_by_count":8,"area":null},{"id":"2512.16762","title":"NRGPT: An Energy-based Alternative for GPT","created":"2025-12-18","cited_by_count":8,"area":null},{"id":"2110.15072","title":"Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces","created":"2021-10-28","cited_by_count":7,"area":null},{"id":"2403.02920","title":"TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax","created":"2024-03-05","cited_by_count":7,"area":null},{"id":"2406.13770","title":"Elliptical Attention","created":"2024-06-19","cited_by_count":7,"area":null},{"id":"2107.12460","title":"Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers","created":"2021-07-26","cited_by_count":6,"area":null},{"id":"2110.04169","title":"Iterative Decoding for Compositional Generalization in Transformers","created":"2021-10-08","cited_by_count":6,"area":null},{"id":"2203.12276","title":"ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention","created":"2022-03-23","cited_by_count":6,"area":null},{"id":"2204.10670","title":"Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention","created":"2022-04-22","cited_by_count":6,"area":null},{"id":"2210.08031","title":"Neural Attentive Circuits","created":"2022-10-14","cited_by_count":6,"area":null},{"id":"2302.08374","title":"Efficiency 360: Efficient Vision Transformers","created":"2023-02-16","cited_by_count":6,"area":null},{"id":"2305.19999","title":"Beam Tree Recursive Cells","created":"2023-05-31","cited_by_count":6,"area":null},{"id":"2405.17534","title":"SMR: State Memory Replay for Long Sequence Modeling","created":"2024-05-27","cited_by_count":6,"area":null},{"id":"2407.08330","title":"HDT: Hierarchical Document Transformer","created":"2024-07-11","cited_by_count":6,"area":null},{"id":"2112.05359","title":"Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences","created":"2021-12-10","cited_by_count":5,"area":null},{"id":"2209.12816","title":"Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers","created":"2022-09-26","cited_by_count":5,"area":null},{"id":"2311.04449","title":"Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability","created":"2023-11-08","cited_by_count":5,"area":null},{"id":"2402.00976","title":"Investigating Recurrent Transformers with Dynamic Halt","created":"2024-02-01","cited_by_count":5,"area":null},{"id":"2405.15310","title":"Spectraformer: A Unified Random Feature Framework for Transformer","created":"2024-05-24","cited_by_count":5,"area":null},{"id":"2406.08128","title":"Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences","created":"2024-06-12","cited_by_count":5,"area":null},{"id":"2210.15541","title":"Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost","created":"2022-10-27","cited_by_count":4,"area":null},{"id":"2211.16368","title":"DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention","created":"2022-11-24","cited_by_count":4,"area":null},{"id":"2302.05400","title":"DNArch: Learning Convolutional Neural Architectures by Backpropagation","created":"2023-02-10","cited_by_count":4,"area":null},{"id":"2307.10779","title":"Efficient Beam Tree Recursion","created":"2023-07-20","cited_by_count":4,"area":null},{"id":"2406.13781","title":"A Primal-Dual Framework for Transformers and Neural Networks","created":"2024-06-19","cited_by_count":4,"area":null},{"id":"2505.08516","title":"Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain","created":"2025-05-13","cited_by_count":4,"area":null},{"id":"2402.07543","title":"Show Me How It's Done: The Role of Explanations in Fine-Tuning Language Models","created":"2024-02-12","cited_by_count":3,"area":null},{"id":"2407.17771","title":"Banyan: Improved Representation Learning with Explicit Structure","created":"2024-07-25","cited_by_count":3,"area":null},{"id":"2108.04962","title":"Adaptive Multi-Resolution Attention with Linear Complexity","created":"2021-08-10","cited_by_count":2,"area":null},{"id":"2109.08184","title":"Sparse Factorization of Large Square Matrices","created":"2021-09-16","cited_by_count":2,"area":null},{"id":"2201.00490","title":"Learning with Latent Structures in Natural Language Processing: A Survey","created":"2022-01-03","cited_by_count":2,"area":null},{"id":"2307.14193","title":"Efficient Learning of Discrete-Continuous Computation Graphs","created":"2023-07-26","cited_by_count":2,"area":null},{"id":"2312.12742","title":"Cached Transformers: Improving Transformers with Differentiable Memory Cache","created":"2023-12-20","cited_by_count":2,"area":null},{"id":"2402.17407","title":"A Neural Rewriting System to Solve Algorithmic Problems","created":"2024-02-27","cited_by_count":2,"area":null},{"id":"2406.00894","title":"Pretrained Hybrids with MAD Skills","created":"2024-06-02","cited_by_count":2,"area":null},{"id":"2410.10914","title":"Towards Better Multi-head Attention via Channel-wise Sample Permutation","created":"2024-10-14","cited_by_count":2,"area":null},{"id":"2509.17186","title":"Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modeling","created":"2025-09-21","cited_by_count":2,"area":null},{"id":"2509.20503","title":"Myosotis: structured computation for attention like layer","created":"2025-09-24","cited_by_count":2,"area":null},{"id":"2104.04998","title":"Unsupervised Learning of Explainable Parse Trees for Improved Generalisation","created":"2021-04-11","cited_by_count":1,"area":null},{"id":"2111.14034","title":"ORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoning","created":"2021-11-28","cited_by_count":1,"area":null},{"id":"2202.11364","title":"FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks","created":"2022-02-23","cited_by_count":1,"area":null},{"id":"2306.12198","title":"Opening the Black Box: Analyzing Attention Weights and Hidden States in Pre-trained Language Models for Non-language Tasks","created":"2023-06-21","cited_by_count":1,"area":null},{"id":"2306.12205","title":"Investigating Pre-trained Language Models on Cross-Domain Datasets, a Step Closer to General AI","created":"2023-06-21","cited_by_count":1,"area":null},{"id":"2310.17683","title":"Sliceformer: Make Multi-head Attention as Simple as Sorting in Discriminative Tasks","created":"2023-10-26","cited_by_count":1,"area":null},{"id":"2408.16261","title":"Evaluating Time-Series Training Dataset through Lens of Spectrum in Deep State Space Models","created":"2024-08-29","cited_by_count":1,"area":null},{"id":"2409.01531","title":"On the Design Space Between Transformers and Recursive Neural Nets","created":"2024-09-03","cited_by_count":1,"area":null},{"id":"2505.12252","title":"SchoenbAt: Rethinking Attention with Polynomial basis","created":"2025-05-18","cited_by_count":1,"area":null},{"id":"2505.18369","title":"Small Models, Smarter Learning: The Power of Joint Task Training","created":"2025-05-23","cited_by_count":1,"area":null},{"id":"2511.14214","title":"Do Large Language Models (LLMs) Understand Chronology?","created":"2025-11-18","cited_by_count":1,"area":null},{"id":"2602.00534","title":"AIRE-Prune: Asymptotic Impulse-Response Energy for State Pruning in State Space Models","created":"2026-01-31","cited_by_count":1,"area":null},{"id":"2605.11855","title":"Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications","created":"2026-05-12","cited_by_count":1,"area":null},{"id":"2605.21070","title":"Towards Understanding Self-Pretraining for Sequence Classification","created":"2026-05-20","cited_by_count":1,"area":null},{"id":"2605.26035","title":"Length Generalization with Log-Depth Recurrent Units","created":"2026-05-25","cited_by_count":1,"area":null},{"id":"2606.27748","title":"Flexformer: Flexible Linear Transformer with Learnable Attention Kernel","created":"2026-06-26","cited_by_count":1,"area":null},{"id":"2206.04806","title":"Syntactic Inductive Biases for Deep Learning Methods","created":"2022-06-08","cited_by_count":0,"area":null}],"next_cursor":"100"}