mlsanityData to 7 Oct 2026

A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks

arXiv 2410.22391 ↗First posted 29 Oct 2024cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

In recent years, there has been a trend in the field of Reinforcement Learning (RL) towards large action models trained offline on large-scale datasets via sequence modeling. Existing models are primarily based on the Transformer architecture, which result in powerful agents. However, due to slow inference times, Transformer-based approaches are impractical for real-time applications, such as robotics. Recently, modern recurrent architectures, such as xLSTM and Mamba, have been proposed that exhibit parallelization benefits during training similar to the Transformer architecture while offering fast inference. In this work, we study the aptitude of these modern recurrent architectures for large action models. Consequently, we propose a Large Recurrent Action Model (LRAM) with an xLSTM at its core that comes with linear-time inference complexity and natural sequence length extrapolation abilities. Experiments on 432 tasks from 6 domains show that LRAM compares favorably to Transformers in terms of performance and speed.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 88 of its 124 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

88 of 124 references.

References · 88

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
38 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  13. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  14. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  15. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  16. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  17. 2017Mixed Precision Trainingcited by 1,135
  18. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  19. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  20. 2018DeepMind Control Suitecited by 935
  21. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  22. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  23. 2022A Generalist Agentcited by 802
  24. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  25. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  26. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  27. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  28. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  29. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  30. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  31. 2020Reinforcement Learning with Augmented Datacited by 452
  32. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  33. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  34. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  35. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  36. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  37. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  38. 2022Multi-Game Decision Transformerscited by 212

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  2. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
  3. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  4. 2025Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernelscited by 12
  5. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 8
  6. 2025Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action modelscited by 6
  7. 2025Optimistic critics can empower small actorscited by 4
  8. 2026Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Taskscited by 3
  9. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  10. 2025pLSTM: parallelizable Linear Source Transition Mark networkscited by 2
  11. 2025A Deep Reinforcement Learning Approach to Automated Stock Trading, using xLSTM Networkscited by 1
  12. 2026Diffusion-Guided Uncertainty-Aware Delayed Policy Optimizationcited by 0
2 more
  1. 2026Operationally Guided Placement-Aware Learning for Industrial Online 3D Bin Packingcited by 0
  2. 2026SmoLSTM: A Compact Vision-Language-Action Model with Recurrent Memory that Persistscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systems
  2. 2026PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models
  3. 2026Reducing Temporal Redundancy for Efficient Vision-Language-Action Inference
  4. 2026rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inference
  5. 2024Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient
  6. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenization
  7. 2026ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
  8. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning