mlsanityData to 7 Oct 2026

Vision-LSTM: xLSTM as Generic Vision Backbone

arXiv 2406.04303 ↗First posted 6 Jun 2024cs.CV cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

Transformers are widely used as generic backbones in computer vision, despite initially introduced for natural language processing. Recently, the Long Short-Term Memory (LSTM) has been extended to a scalable and performant architecture - the xLSTM - which overcomes long-standing LSTM limitations via exponential gating and parallelizable matrix memory structure. In this report, we introduce Vision-LSTM (ViL), an adaption of the xLSTM building blocks to computer vision. ViL comprises a stack of xLSTM blocks where odd blocks process the sequence of patch tokens from top to bottom while even blocks go from bottom to top. Experiments show that ViL holds promise to be further deployed as new generic backbone for computer vision architectures.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 72 of its 97 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

72 of 97 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Segment Anythingcited by 8,158
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2016Densely Connected Convolutional Networkscited by 6,691
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2016Layer Normalizationcited by 5,579
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  13. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  14. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  16. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  17. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  18. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  19. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  20. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  21. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  22. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  23. 2019Natural Adversarial Examplescited by 1,401
  24. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  25. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  26. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  27. 2016Colorful Image Colorizationcited by 1,113
  28. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  29. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  30. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  31. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  32. 2016Deep Networks with Stochastic Depthcited by 973
  33. 2024VMamba: Visual State Space Modelcited by 955
  34. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
  35. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  36. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  37. 2021Going deeper with Image Transformerscited by 759
  38. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 23; 2026: 7.

51 papers.

Sort:

By area:

  1. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  2. 2024Quantum Kernel-Based Long Short-term Memory for Climate Time-Series Forecastingcited by 17
  3. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  4. 2024A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyondcited by 16
  5. 2024MobileMamba: Lightweight Multi-Receptive Visual Mamba Networkcited by 16
  6. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  7. 2024Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequencescited by 13
  8. 2024xLSTM-UNet can be an Effective 2D & 3D Medical Image Segmentation Backbone with Vision-LSTM (ViL) better than its Mamba Counterpartcited by 12
  9. 2025Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernelscited by 12
  10. 2025BenchECG and xECG: a benchmark and baseline for ECG foundation modelscited by 8
  11. 2024Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Modelscited by 7
  12. 2024ChangeMinds: Multi-task Framework for Detecting and Describing Changes in Remote Sensingcited by 7
38 more
  1. 2025Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge resultscited by 7
  2. 2024Seg-LSTM: Performance of xLSTM for Semantic Segmentation of Remotely Sensed Imagescited by 5
  3. 2024XLSTM-HVED: Cross-Modal Brain Tumor Segmentation and MRI Reconstruction Method Using Vision XLSTM and Heteromodal Variational Encoder-Decodercited by 5
  4. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Modelscited by 5
  5. 2025xLSTMAD: A Powerful xLSTM-based Method for Anomaly Detectioncited by 5
  6. 2024CDXLSTM: Boosting Remote Sensing Change Detection with Extended Long Short-Term Memorycited by 4
  7. 2025xLSTM-SENet: xLSTM for Single-Channel Speech Enhancementcited by 4
  8. 2025Parallel Sequence Modeling via Generalized Spatial Propagation Networkcited by 4
  9. 2025DKT2: Revisiting Applicable and Comprehensive Knowledge Tracing in Large-Scale Datacited by 4
  10. 2025Linear Attention for Efficient Bidirectional Sequence Modelingcited by 4
  11. 2024Are Vision xLSTM Embedded UNet More Reliable in Medical 3D Image Segmentation?cited by 3
  12. 2024Enhancing Environmental Monitoring through Multispectral Imaging: The WasteMS Dataset for Semantic Segmentation of Lakeside Wastecited by 3
  13. 2025Learning to Learn with Quantum Optimization via Quantum Neural Networkscited by 3
  14. 2025UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNscited by 3
  15. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  16. 2025M3-AGIQA: Multimodal, Multi-Round, Multi-Aspect AI-Generated Image Quality Assessmentcited by 2
  17. 2025pLSTM: parallelizable Linear Source Transition Mark networkscited by 2
  18. 2025MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancementcited by 2
  19. 2025Plug In, Grade Right: Psychology-Inspired AGIQAcited by 2
  20. 2024xLSTM-FER: Enhancing Student Expression Recognition with Extended Vision Long Short-Term Memory Networkcited by 1
  21. 2024Enhancing Autism Spectrum Disorder Early Detection with the Parent-Child Dyads Block-Play Protocol and an Attention-enhanced GCN-xLSTM Hybrid Deep Learning Frameworkcited by 0
  22. 2024What Makes a Face Look like a Hat: Decoupling Low-level and High-level Visual Properties with Image Tripletscited by 0
  23. 2024When Mamba Meets xLSTM: An Efficient and Precise Method with the xLSTM-VMUNet Model for Skin lesion Segmentationcited by 0
  24. 2024MSCrackMamba: Leveraging Vision Mamba for Crack Detection in Fused Multispectral Imagerycited by 0
  25. 2024MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performancecited by 0
  26. 2024BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Modelscited by 0
  27. 2025UNetVL: Enhancing 3D Medical Image Segmentation with Chebyshev KAN Powered Vision-LSTMcited by 0
  28. 2025Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structurescited by 0
  29. 2025InceptionMamba: An Efficient Hybrid Network with Large Band Convolution and Bottleneck Mambacited by 0
  30. 2025Atmos-Bench: 3D Atmospheric Structures for Climate Insightcited by 0
  31. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  32. 2025TM-UNet: Token-Memory Enhanced Sequential Modeling for Efficient Medical Image Segmentationcited by 0
  33. 2025GSPN-2: Efficient Parallel Sequence Modelingcited by 0
  34. 2026OSA: Echocardiography Video Segmentation via Orthogonalized State Update and Anatomical Prior-aware Feature Enhancementcited by 0
  35. 2026FACT: A Simple and Efficient Framework for Active Finetuningcited by 0
  36. 2026Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformerscited by 0
  37. 2026Diffusion-Guided Uncertainty-Aware Delayed Policy Optimizationcited by 0
  38. 2026Visible-Light Imaging Diagnosis of Neutral Particle Emission Tomography in the Tokamak Divertor: An Efficient Transformer-based Surrogate Modelcited by 0

50 most cited of 51 All 51 (JSON)