mlsanityData to 14 Oct 2026

Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning

arXiv 2605.31261 ↗First posted 29 May 2026cs.LG cs.AI stat.MLcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The family of linear recurrent neural networks has shown strong performance as recurrent memory units in partially observable reinforcement learning. We provide a theoretical justification for their empirical effectiveness by constructing and studying two linear filters: (i) the first exactly reproduces the pre-softmax logits of the belief vector in a hidden Markov model (HMM) under a deterministic transition matrix, thereby serving as a sufficient statistic for optimal policy learning, (ii) the second achieves vanishing state-decoding error under a nearly deterministic transition matrix, thus reducing state ambiguity to near zero. The results extend to action-controlled HMMs, where the corresponding linear filters become time-varying with action-dependent dynamics. We illustrate our main results through numerical experiments and further show that the constructed linear filter serves as a strong feature extractor in a small reinforcement learning game.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 35 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

19 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning4 May
  2. 2025Partially Observable Reinforcement Learning with Memory Traces19 Mar
  3. 2024Rethinking Transformers in Solving POMDPs27 May

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  7. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  8. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  9. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  10. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  11. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  12. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
4 more
  1. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 62
  2. 2022Provable Reinforcement Learning with a Short-Term Memorycited by 40
  3. 2022Finite-Time Analysis of Natural Actor-Critic for POMDPscited by 13
  4. 2022Optimal Aggregation Strategies for Social Learning over Graphscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Does Learning Recurrent Networks Matter in Partially Observable RL?
  2. 2024Recurrent Reinforcement Learning with Memoroids
  3. 2026Belief-State RWKV for Reinforcement Learning under Partial Observability
  4. 2025Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes
  5. 2024Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability
  6. 2025Zero-Shot Reinforcement Learning Under Partial Observability
  7. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
  8. 2018Learning Finite State Representations of Recurrent Policy Networks