mlsanityData to 7 Oct 2026

Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation

arXiv 2609.35333 ↗First posted 28 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Algorithm Distillation (AD) has demonstrated the remarkable ability of Transformers to perform in-context reinforcement learning without explicit weight updates. However, capturing long-term learning progress necessitates expansive context windows, which incur prohibitive memory costs and limit scalability in complex, long-horizon tasks. To address this bottleneck, we propose Recurrent Algorithm Distillation (RAD). RAD employs a dual-component architecture: a Compression Transformer that distills extended interaction histories into compact latent tokens, and an AD Transformer that auto-regressively generates actions using a hybrid context of these compressed memories and recent transitions. By maintaining a fixed-size latent buffer, RAD decouples the effective history length from computational complexity, functionally providing the model with a long-horizon memory. Empirical evaluations across diverse environments demonstrate that RAD matches the asymptotic performance of standard AD with significantly reduced context window sizes, offering a scalable solution for efficient in-context decision-making.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 53 of its 60 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

53 of 60 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2018On First-Order Meta-Learning Algorithmscited by 1,003
38 more
  1. 2022A Generalist Agentcited by 802
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  4. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  5. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  6. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  7. 2016Learning to reinforcement learncited by 369
  8. 2024Titans: Learning to Memorize at Test Timecited by 303
  9. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  10. 2022Memorizing Transformerscited by 219
  11. 2022Multi-Game Decision Transformerscited by 212
  12. 2022Recurrent Memory Transformercited by 197
  13. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  15. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  16. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  17. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  18. 2022Block-Recurrent Transformerscited by 107
  19. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  20. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  21. 2023Elastic Decision Transformercited by 66
  22. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  23. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  24. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  25. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 37
  26. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  27. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 33
  28. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 26
  29. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 25
  30. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 23
  31. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  32. 2023Recurrent Action Transformer with Memorycited by 17
  33. 2025Context Cascade Compression: Exploring the Upper Limits of Text Compressioncited by 17
  34. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 12
  35. 2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
  36. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  37. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  38. 2025Scalable In-Context Q-Learningcited by 5

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.