mlsanityData to 7 Oct 2026

MaskMamba: A Hybrid Mamba-Transformer Model for Masked Image Generation

arXiv 2409.19937 ↗First posted 30 Sep 2024cs.CVcited by 7

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

Abstract (arXiv)

Image generation models have encountered challenges related to scalability and quadratic complexity, primarily due to the reliance on Transformer-based backbones. In this study, we introduce MaskMamba, a novel hybrid model that combines Mamba and Transformer architectures, utilizing Masked Image Modeling for non-autoregressive image synthesis. We meticulously redesign the bidirectional Mamba architecture by implementing two key modifications: (1) replacing causal convolutions with standard convolutions to better capture global context, and (2) utilizing concatenation instead of multiplication, which significantly boosts performance while accelerating inference speed. Additionally, we explore various hybrid schemes of MaskMamba, including both serial and grouped parallel arrangements. Furthermore, we incorporate an in-context condition that allows our model to perform both class-to-image and text-to-image generation tasks. Our MaskMamba outperforms Mamba-based and Transformer-based models in generation quality. Notably, it achieves a remarkable 54.44% improvement in inference speed at a resolution of 2048× 2048 over Transformer.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 44 of its 54 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

44 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generation24 May
  2. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis23 May

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
30 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  12. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  13. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  14. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  15. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  16. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  17. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  18. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  19. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  20. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  21. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  22. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  23. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  24. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  25. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  26. 2023Diffusion Models Without Attentioncited by 75
  27. 2022Improved Masked Image Generation with Token-Criticcited by 63
  28. 2023Block-State Transformerscited by 24
  29. 2024Revisiting Non-Autoregressive Transformers for Efficient Image Synthesiscited by 22
  30. 2024Uncovering Selective State Space Model's Capabilities in Lifelong Sequential Recommendationcited by 17

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 5.

7 papers.

  1. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  2. 2025DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizercited by 13
  3. 2024MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretrainingcited by 12
  4. 2025Small Vision-Language Models: A Survey on Compact Architectures and Techniquescited by 4
  5. 2025PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applicationscited by 1
  6. 2025D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokenscited by 0
  7. 2025HybridTM: Combining Transformer and Mamba for 3D Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation
  2. 2024Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer
  3. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis
  4. 2024Scalable Autoregressive Image Generation with Mamba
  5. 2024Dimba: Transformer-Mamba Diffusion Models
  6. 2025Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generation
  7. 2024MaskBit: Embedding-free Image Generation via Bit Tokens
  8. 2025SISMA: Semantic Face Image Synthesis with Mamba