mlsanityData to 7 Oct 2026

Block-State Transformers

arXiv 2306.09539 ↗First posted 15 Jun 2023cs.CL cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

State space models (SSMs) have shown impressive results on tasks that require modeling long-range dependencies and efficiently scale to long sequences owing to their subquadratic runtime complexity. Originally designed for continuous signals, SSMs have shown superior performance on a plethora of tasks, in vision and audio; however, SSMs still lag Transformer performance in Language Modeling tasks. In this work, we propose a hybrid layer named Block-State Transformer (BST), that internally combines an SSM sublayer for long-range contextualization, and a Block Transformer sublayer for short-term representation of sequences. We study three different, and completely parallelizable, variants that integrate SSMs and block-wise attention. We show that our model outperforms similar Transformer-based architectures on language modeling perplexity and generalizes to longer sequences. In addition, the Block-State Transformer demonstrates more than tenfold increase in speed at the layer level compared to the Block-Recurrent Transformer when model parallelization is employed.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

32 of 46 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
20 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  6. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  7. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  8. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  9. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  10. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  11. 2022Long Range Language Modeling via Gated State Spacescited by 265
  12. 2020Sparse Sinkhorn Attentioncited by 250
  13. 2022Memorizing Transformerscited by 219
  14. 2022Transformer Quality in Linear Timecited by 189
  15. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  16. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  17. 2022Block-Recurrent Transformerscited by 107
  18. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  19. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  20. 2020Multi-scale Transformer Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 13; 2025: 4; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2023Efficient Large Language Models: A Surveycited by 183
  6. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  7. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  8. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  9. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  10. 2024A Survey of Mambacited by 39
  11. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  12. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
12 more
  1. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  2. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  3. 2025Generalizable, real-time neural decoding with hybrid state-space modelscited by 8
  4. 2024MaskMamba: A Hybrid Mamba-Transformer Model for Masked Image Generationcited by 7
  5. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  6. 2026Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behaviorcited by 5
  7. 2025TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embeddingcited by 3
  8. 2026Maglev: Sliding Recurrent Memorycited by 3
  9. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  10. 2026MambaFormer: Token-Level Guided Routing Mixture-of-Experts for Accurate and Efficient Clinical Assistancecited by 1
  11. 2026MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deploymentcited by 1
  12. 2024DualKanbaFormer: An Efficient Selective Sparse Framework for Multimodal Aspect-based Sentiment Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
  2. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  3. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
  5. 2023Multi-Head State Space Model for Speech Recognition
  6. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inference
  7. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  8. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNs