mlsanityData to 14 Oct 2026

Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning

arXiv 2605.16318 ↗First posted 4 May 2026cs.LGcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Building and maintaining state to learn policies and value functions is critical for deploying reinforcement learning (RL) agents in the real world. Recurrent neural networks (RNNs) have become a key point of interest for the state-building problem, and several large-scale reinforcement learning agents incorporate recurrent networks. While RNNs have become a mainstay in many RL applications, many key design choices and implementation details responsible for performance improvements are often not reported. In this work, we discuss one axis on which RNN architectures can be (and have been) modified for use in RL. Specifically, we look at how action information can be incorporated into the state update function of a recurrent cell. We discuss several choices in using action information and empirically evaluate the resulting architectures on a set of illustrative domains. Finally, we discuss future work in developing recurrent cells and discuss challenges specific to the RL setting.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 74 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

32 of 74 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  9. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  10. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  11. 2015Visualizing and Understanding Recurrent Networkscited by 160
  12. 2018Deep Variational Reinforcement Learning for POMDPscited by 125
20 more
  1. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  2. 2015Memory-based control with recurrent neural networkscited by 78
  3. 2018On Improving Deep Reinforcement Learning for POMDPscited by 50
  4. 2017Unbiased Online Recurrent Optimizationcited by 47
  5. 2018Don't Unroll Adjoint: Differentiating SSA-Form Programscited by 45
  6. 2017Understanding Hidden Memories of Recurrent Neural Networkscited by 44
  7. 2017English Conversational Telephone Speech Recognition by Humans and Machinescited by 40
  8. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 34
  9. 2018Approximating Real-Time Recurrent Learning with Random Kronecker Factorscited by 29
  10. 2020Expected Eligibility Tracescited by 27
  11. 2020A Practical Sparse Approximation for Real Time Recurrent Learningcited by 18
  12. 2018General Value Function Networkscited by 16
  13. 2016On Multiplicative Integration with Recurrent Neural Networkscited by 15
  14. 2019Towards Non-saturating Recurrent Units for Modelling Long-term Dependenciescited by 15
  15. 2017Predictive State Recurrent Neural Networkscited by 14
  16. 2015Temporal-Difference Networkscited by 13
  17. 2020From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learningcited by 13
  18. 2015Larger-Context Language Modellingcited by 10
  19. 2015Learning to Predict Independent of Spancited by 5
  20. 2021Learning Agent State Online with Recurrent Generate-and-Testcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learning
  2. 2025RLBenchNet: The Right Network for the Right Reinforcement Learning Task
  3. 2026Does Learning Recurrent Networks Matter in Partially Observable RL?
  4. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
  5. 2018Learning Finite State Representations of Recurrent Policy Networks
  6. 2024Recurrent Reinforcement Learning with Memoroids
  7. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
  8. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models