mlsanityData to 14 Oct 2026

Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow

arXiv 2512.20513 ↗First posted 23 Dec 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recurrent off-policy deep reinforcement learning models achieve state-of-the-art performance but are often sidelined due to their high computational demands. In response, we introduce RISE (Recurrent Integration via Simplified Encodings), a novel approach that can leverage recurrent networks in any image-based off-policy RL setting without significant computational overheads via using both learnable and non-learnable encoder layers. When integrating RISE into leading non-recurrent off-policy RL algorithms, we observe a 35.6% human-normalized interquartile mean (IQM) performance improvement across the Atari benchmark. We analyze various implementation strategies to highlight the versatility and potential of our proposed framework.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 50 of its 60 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

50 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC6 Nov
  2. 2024Simplifying Deep Temporal Difference Learning5 Jul
  3. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate24 May
  4. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data1 Mar
  5. 2024Recurrent Reinforcement Learning with Memoroids15 Feb

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2015Continuous control with deep reinforcement learningcited by 3,407
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
33 more
  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  3. 2023Mastering Diverse Domains through World Modelscited by 1,253
  4. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  11. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  13. 2015Attention-Based Models for Speech Recognitioncited by 401
  14. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  15. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  16. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  17. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  18. 2017Noisy Networks for Explorationcited by 266
  19. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  20. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  21. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  22. 2020Phasic Policy Gradientcited by 123
  23. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  24. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  25. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  26. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  27. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  28. 2023Small batch deep reinforcement learningcited by 36
  29. 2022Human-level Atari 200x fastercited by 28
  30. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
  31. 2021Fast and Data-Efficient Training of Rainbow: an Experimental Study on Ataricited by 10
  32. 2023Reinforcement Learning with Fast and Forgetful Memorycited by 9
  33. 2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  2. 2021A Minimalist Approach to Offline Reinforcement Learning
  3. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  4. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models
  5. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  6. 2022Boosting Offline Reinforcement Learning via Data Rebalancing
  7. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
  8. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards