{"list":"other","total":69,"rows":[{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2309.06180","title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","created":"2023-09-12","cited_by_count":5944},{"id":"1803.05457","title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","created":"2018-03-14","cited_by_count":4886},{"id":"2104.09864","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","created":"2021-04-20","cited_by_count":4703},{"id":"1905.07830","title":"HellaSwag: Can a Machine Really Finish Your Sentence?","created":"2019-05-19","cited_by_count":4100},{"id":"2205.14135","title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","created":"2022-05-27","cited_by_count":3673},{"id":"1911.11641","title":"PIQA: Reasoning about Physical Commonsense in Natural Language","created":"2019-11-26","cited_by_count":2883},{"id":"2307.08691","title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","created":"2023-07-17","cited_by_count":2598},{"id":"1711.05101","title":"Decoupled Weight Decay Regularization","created":"2017-11-14","cited_by_count":1831},{"id":"2308.14508","title":"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding","created":"2023-08-28","cited_by_count":1417},{"id":"2404.06654","title":"RULER: What's the Real Context Size of Your Long-Context Language Models?","created":"2024-04-09","cited_by_count":1346},{"id":"2405.21060","title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","created":"2024-05-31","cited_by_count":1336},{"id":"2406.17557","title":"The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale","created":"2024-06-25","cited_by_count":1160},{"id":"1803.02155","title":"Self-Attention with Relative Position Representations","created":"2018-03-06","cited_by_count":1120},{"id":"2108.12409","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","created":"2021-08-27","cited_by_count":1103},{"id":"1606.06031","title":"The LAMBADA dataset: Word prediction requiring a broad discourse context","created":"2016-06-20","cited_by_count":992},{"id":"2306.15595","title":"Extending Context Window of Large Language Models via Positional Interpolation","created":"2023-06-27","cited_by_count":740},{"id":"1911.05507","title":"Compressive Transformers for Long-Range Sequence Modelling","created":"2019-11-13","cited_by_count":726},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"1712.07040","title":"The NarrativeQA Reading Comprehension Challenge","created":"2017-12-19","cited_by_count":597},{"id":"2412.06464","title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","created":"2024-12-09","cited_by_count":532},{"id":"2407.08608","title":"FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision","created":"2024-07-11","cited_by_count":519},{"id":"2312.06635","title":"Gated Linear Attention Transformers with Hardware-Efficient Training","created":"2023-12-11","cited_by_count":514},{"id":"2102.11174","title":"Linear Transformers Are Secretly Fast Weight Programmers","created":"2021-02-22","cited_by_count":502},{"id":"2310.01889","title":"Ring Attention with Blockwise Transformers for Near-Infinite Context","created":"2023-10-03","cited_by_count":457},{"id":"2406.06484","title":"Parallelizing Linear Transformers with the Delta Rule over Sequence Length","created":"2024-06-10","cited_by_count":396},{"id":"2210.10749","title":"Transformers Learn Shortcuts to Automata","created":"2022-10-19","cited_by_count":268},{"id":"2402.01032","title":"Repeat After Me: Transformers are Better than State Space Models at Copying","created":"2024-02-01","cited_by_count":216},{"id":"2406.10149","title":"BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack","created":"2024-06-14","cited_by_count":215},{"id":"2402.18668","title":"Simple linear attention language models balance the recall-throughput tradeoff","created":"2024-02-28","cited_by_count":208},{"id":"2312.04927","title":"Zoology: Measuring and Improving Recall in Efficient Language Models","created":"2023-12-08","cited_by_count":204},{"id":"2006.15595","title":"Rethinking Positional Encoding in Language Pre-training","created":"2020-06-28","cited_by_count":197},{"id":"2404.08819","title":"The Illusion of State in State-Space Models","created":"2024-04-12","cited_by_count":187},{"id":"1805.02867","title":"Online normalizer calculation for softmax","created":"2018-05-08","cited_by_count":162},{"id":"2112.05682","title":"Self-attention Does Not Need $O(n^2)$ Memory","created":"2021-12-10","cited_by_count":154},{"id":"2402.14811","title":"Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking","created":"2024-02-22","cited_by_count":149},{"id":"2503.14456","title":"RWKV-7 \"Goose\" with Expressive Dynamic State Evolution","created":"2025-03-18","cited_by_count":141},{"id":"2105.13120","title":"Sequence Parallelism: Long Sequence Training from System Perspective","created":"2021-05-26","cited_by_count":126},{"id":"2310.05209","title":"Scaling Laws of RoPE-based Extrapolation","created":"2023-10-08","cited_by_count":125},{"id":"2102.11090","title":"Position Information in Transformers: An Overview","created":"2021-02-22","cited_by_count":103},{"id":"2411.12537","title":"Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues","created":"2024-11-19","cited_by_count":101},{"id":"2402.09371","title":"Transformers Can Achieve Length Generalization But Not Robustly","created":"2024-02-14","cited_by_count":86},{"id":"2306.00946","title":"Exposing Attention Glitches with Flip-Flop Language Modeling","created":"2023-06-01","cited_by_count":75},{"id":"2402.19047","title":"Theoretical Foundations of Deep Selective State-Space Models","created":"2024-02-29","cited_by_count":74},{"id":"2009.13658","title":"Improve Transformer Models with Better Relative Position Embeddings","created":"2020-09-28","cited_by_count":73},{"id":"2210.02671","title":"A Logic for Expressing Log-Precision Transformers","created":"2022-10-06","cited_by_count":67},{"id":"2110.07732","title":"The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization","created":"2021-10-14","cited_by_count":62},{"id":"2405.18719","title":"Contextual Position Encoding: Learning to Count What's Important","created":"2024-05-29","cited_by_count":54},{"id":"2503.02130","title":"Forgetting Transformer: Softmax Attention with a Forget Gate","created":"2025-03-03","cited_by_count":53},{"id":"2504.02807","title":"MegaMath: Pushing the Limits of Open Math Corpora","created":"2025-04-03","cited_by_count":51},{"id":"2305.02363","title":"Entity Tracking in Language Models","created":"2023-05-03","cited_by_count":44},{"id":"2105.08399","title":"Relative Positional Encoding for Transformers with Linear Complexity","created":"2021-05-18","cited_by_count":37},{"id":"2405.14722","title":"DAPE: Data-Adaptive Positional Encoding for Length Extrapolation","created":"2024-05-23","cited_by_count":31},{"id":"2406.09347","title":"Separations in the Representational Capabilities of Transformers and Recurrent Architectures","created":"2024-06-13","cited_by_count":29},{"id":"2410.02703","title":"Selective Attention Improves Transformer","created":"2024-10-03","cited_by_count":26},{"id":"2411.07602","title":"Circuit Complexity Bounds for RoPE-based Transformer Architecture","created":"2024-11-12","cited_by_count":26},{"id":"2505.03005","title":"RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale","created":"2025-05-05","cited_by_count":24},{"id":"2502.20129","title":"Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking","created":"2025-02-27","cited_by_count":18},{"id":"2306.04640","title":"ModuleFormer: Modularity Emerges from Mixture-of-Experts","created":"2023-06-07","cited_by_count":17},{"id":"2307.09270","title":"Linearized Relative Positional Encoding","created":"2023-07-18","cited_by_count":17},{"id":"2412.19350","title":"On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages","created":"2024-12-26","cited_by_count":15},{"id":"2410.17980","title":"Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study","created":"2024-10-23","cited_by_count":13},{"id":"2312.16045","title":"Algebraic Positional Encodings","created":"2023-12-26","cited_by_count":10},{"id":"2410.04798","title":"DAPE V2: Process Attention Score as Feature Map for Length Extrapolation","created":"2024-10-07","cited_by_count":10},{"id":"2009.13977","title":"What if Neural Networks had SVDs?","created":"2020-09-29","cited_by_count":8},{"id":"2410.05565","title":"Chain and Causal Attention for Efficient Entity Tracking","created":"2024-10-07","cited_by_count":8},{"id":"2504.06949","title":"Adaptive Computation Pruning for the Forgetting Transformer","created":"2025-04-09","cited_by_count":5},{"id":"2503.08067","title":"Context-aware Biases for Length Extrapolation","created":"2025-03-11","cited_by_count":2}],"next_cursor":null}