mlsanityData to 14 Oct 2026

Scaling Algorithm Distillation for Continuous Control with Mamba

arXiv 2506.13892 ↗First posted 16 Jun 2025cs.LG cs.AI cs.ROcited by 2

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

also in 1 more
Abstract (arXiv)

Algorithm Distillation (AD) was recently proposed as a new approach to perform In-Context Reinforcement Learning (ICRL) by modeling across-episodic training histories autoregressively with a causal transformer model. However, due to practical limitations induced by the attention mechanism, experiments were bottlenecked by the transformer's quadratic complexity and limited to simple discrete environments with short time horizons. In this work, we propose leveraging the recently proposed Selective Structured State Space Sequence (S6) models, which achieved state-of-the-art (SOTA) performance on long-range sequence modeling while scaling linearly in sequence length. Through four complex and continuous Meta Reinforcement Learning environments, we demonstrate the overall superiority of Mamba, a model built with S6 layers, over a transformer model for AD. Additionally, we show that scaling AD to very long contexts can improve ICRL performance and make it competitive even with a SOTA online meta RL baseline.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 33 of its 36 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

33 of 36 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Segment Anythingcited by 8,234
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
21 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  5. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  7. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  10. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  12. 2016Learning to reinforcement learncited by 377
  13. 2024Many-Shot In-Context Learningcited by 206
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  15. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  16. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  17. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  18. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  19. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  20. 2019Meta-Q-Learningcited by 62
  21. 2024Is Mamba Capable of In-Context Learning?cited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026In-Context Reinforcement Learning under Non-Stationarity: A Surveycited by 1
  2. 2026MATE: Solving Contextual Markov Decision Processes with Memory of Accumulated Transition Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient
  2. 2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling
  3. 2024Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning
  4. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces
  5. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation
  6. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
  7. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?
  8. 2025Vintix: Action Model via In-Context Reinforcement Learning