mlsanityData to 14 Oct 2026

Does Learning Recurrent Networks Matter in Partially Observable RL?

arXiv 2610.13197 ↗First posted 9 Oct 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at https://github.com/Aphlios/FRN-POMDP

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 66 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

38 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Investigating Memory in Model-Free RL with POPGym Arcade3 Mar
  2. 2024Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks17 Dec
  3. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning14 Oct
  4. 2024Recurrent Reinforcement Learning with Memoroids15 Feb

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2016Layer Normalizationcited by 5,669
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  12. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
22 more
  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  2. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  3. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  4. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  5. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  6. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 148
  7. 2019Behaviour Suite for Reinforcement Learningcited by 132
  8. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  9. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  10. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
  11. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 62
  12. 2018Unsupervised Predictive Memory in a Goal-Directed Agentcited by 55
  13. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 54
  14. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 40
  15. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 36
  16. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 36
  17. 2021CoBERL: Contrastive BERT for Reinforcement Learningcited by 21
  18. 2020On the stability properties of Gated Recurrent Units neural networkscited by 19
  19. 2020Reservoir Computing meets Recurrent Kernels and Structured Transformscited by 16
  20. 2023Reinforcement Learning with Fast and Forgetful Memorycited by 9
  21. 2023AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learningcited by 6
  22. 2026ParalESN: Enabling parallel information processing in Reservoir Computingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning
  2. 2025Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes
  3. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
  4. 2026Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
  5. 2024Rethinking Transformers in Solving POMDPs
  6. 2026Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
  7. 2025Zero-Shot Reinforcement Learning Under Partial Observability
  8. 2026Belief-State RWKV for Reinforcement Learning under Partial Observability