mlsanityData to 14 Oct 2026

Stochastic Layer-Wise Shuffle for Improving Vision Mamba Training

arXiv 2408.17081 ↗First posted 30 Aug 2024cs.CVcited by 0

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

Abstract (arXiv)

Recent Vision Mamba (Vim) models exhibit nearly linear complexity in sequence length, making them highly attractive for processing visual data. However, the training methodologies and their potential are still not sufficiently explored. In this paper, we investigate strategies for Vim and propose Stochastic Layer-Wise Shuffle (SLWS), a novel regularization method that can effectively improve the Vim training. Without architectural modifications, this approach enables the non-hierarchical Vim to get leading performance on ImageNet-1K compared with the similar type counterparts. Our method operates through four simple steps per layer: probability allocation to assign layer-dependent shuffle rates, operation sampling via Bernoulli trials, sequence shuffling of input tokens, and order restoration of outputs. SLWS distinguishes itself through three principles: textit(1) Plug-and-play: No architectural modifications are needed, and it is deactivated during inference. textit(2) Simple but effective: The four-step process introduces only random permutations and negligible overhead. textit(3) Intuitive design: Shuffling probabilities grow linearly with layer depth, aligning with the hierarchical semantic abstraction in vision models. Our work underscores the importance of tailored training strategies for Vim models and provides a helpful way to explore their scalability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 82 of its 101 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

82 of 101 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining31 Jul
  2. 2024Autoregressive Pretraining with Mamba in Vision11 Jun
  3. 2024Vim-F: Visual State Space Model Benefiting from Learning in the Frequency Domain29 May
  4. 2024Mamba-R: Vision Mamba ALSO Needs Registers23 May
  5. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition26 Mar
  6. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba15 Mar
  7. 2024LocalMamba: Visual State Space Model with Windowed Selective Scan14 Mar
  8. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model17 Jan

Other references · 74

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017Mask R-CNNcited by 7,277
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  2. 2017Squeeze-and-Excitation Networkscited by 4,789
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  13. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  14. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  15. 2018Group Normalizationcited by 1,524
  16. 2019Root Mean Square Layer Normalizationcited by 1,480
  17. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  18. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  19. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  20. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  21. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  22. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  23. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  24. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  25. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  26. 2016Deep Networks with Stochastic Depthcited by 979
  27. 2024VMamba: Visual State Space Modelcited by 969
  28. 2020Designing Network Design Spacescited by 960
  29. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  30. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  31. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  32. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  33. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  34. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  35. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  36. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  37. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  38. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527

50 most cited of 74 All 74 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamic Vision Mamba
  2. 2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusion
  3. 2025ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mamba
  4. 2026Beyond ZOH: Advanced Discretization Strategies for Vision Mamba
  5. 2025Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing
  6. 2024MambaOut: Do We Really Need Mamba for Vision?
  7. 2025Post-Training Quantization for Vision Mamba with k-Scaled Quantization and Reparameterization
  8. 2024Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion