mlsanityData to 14 Oct 2026

Pretraining Without Attention

arXiv 2212.10544 ↗First posted 20 Dec 2022cs.CL cs.LGcited by 51

Not on the map. Why?

Cited by 20 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have been essential to pretraining success in NLP. While other architectures have been used, downstream accuracy is either significantly worse, or requires attention layers to match standard benchmarks such as GLUE. This work explores pretraining without attention by using recent advances in sequence routing based on state-space models (SSMs). Our proposed model, Bidirectional Gated SSM (BiGS), combines SSM layers with a multiplicative gating architecture that has been effective in simplified sequence modeling architectures. The model learns static layers that do not consider pair-wise interactions. Even so, BiGS is able to match BERT pretraining accuracy on GLUE and can be extended to long-form pretraining of 4096 tokens without approximation. Analysis shows that while the models have similar average accuracy, the approach has different inductive biases than BERT in terms of interactions and syntactic representations. All models from this work are available at https://github.com/jxiw/BiGS.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

35 of 43 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  12. 2018Deep contextualized word representationscited by 2,290
23 more
  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2018Neural Network Acceptability Judgmentscited by 958
  3. 2020Efficient Transformers: A Surveycited by 790
  4. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  5. 2016Language Modeling with Gated Convolutional Networkscited by 702
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  7. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  8. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  10. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  11. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  12. 2022Long Range Language Modeling via Gated State Spacescited by 265
  13. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  14. 2019Assessing BERT's Syntactic Abilitiescited by 192
  15. 2022Transformer Quality in Linear Timecited by 191
  16. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  17. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  18. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  19. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  20. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  21. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  22. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  23. 2021How to Train BERT with an Academic Budgetcited by 80

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 33; 2025: 6; 2026: 2.

51 papers.

Sort:

By area:

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  3. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  6. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  7. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  8. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
  9. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 112
  10. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  11. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  12. 2023Diffusion Models Without Attentioncited by 75
38 more
  1. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  2. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  3. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  4. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 59
  5. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 44
  6. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  7. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 39
  8. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 36
  9. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 33
  10. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  11. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  12. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 21
  13. 2024SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spacescited by 13
  14. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  15. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  16. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12
  17. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  18. 2023On the Long Range Abilities of Transformerscited by 10
  19. 2024A Mamba Foundation Model for Time Series Forecastingcited by 9
  20. 2024LOCOST: State-Space Models for Long Document Abstractive Summarizationcited by 8
  21. 2024Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Modelscited by 8
  22. 2024Serpent: Scalable and Efficient Image Restoration via Multi-scale Structured State Space Modelscited by 7
  23. 2024Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learningcited by 7
  24. 2025EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wildcited by 7
  25. 2024Orchid: Flexible and Data-Dependent Convolution for Sequence Modelingcited by 6
  26. 2024HumMUSS: Human Motion Understanding using State Space Modelscited by 6
  27. 2023Multi-Dimensional Hyena for Spatial Inductive Biascited by 5
  28. 2025Trellis: Learning to Compress Key-Value Memory in Attention Modelscited by 5
  29. 2024The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labelscited by 3
  30. 2024S3Attention: Improving Long Sequence Attention with Smoothed Skeleton Sketchingcited by 2
  31. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  32. 2024Seeing the roads through the trees: A benchmark for modeling spatial dependencies with aerial imagerycited by 1
  33. 2024On the low-shot transferability of [V]-Mambacited by 1
  34. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1
  35. 2025LBMamba: Locally Bi-directional Mambacited by 1
  36. 2025Design Principles for Sequence Models via Coefficient Dynamicscited by 1
  37. 2026Towards Understanding What State Space Models Learn About Codecited by 1
  38. 2025Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusioncited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Token Dropping for Efficient BERT Pretraining
  2. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Models
  3. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
  4. 2025Sliding Window Attention Training for Efficient Large Language Models
  5. 2023Just CHOP: Embarrassingly Simple LLM Compression
  6. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
  7. 2023PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models
  8. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretraining