Reinforcement Learning with Complex (valued) Memories
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from observations and maintain a memory to make effective decisions. While there exist many approaches ranging from gated recurrence to attention mechanisms and model-based RL, the search for effective representational techniques that can capture long-term dependencies remains an active area of research. In this work we revisit Unitary recurrent networks (uRNNs) [Arjovsky et al., 2016, Jing et al., 2017], that demonstrated superior gradient flow and associative recall, expressing the recurrence and the hidden state in a complex vector space. Their norm preserving unitary dynamics enable information propagation through long sequences. To this end, we propose three different versions of uRNNs as drop-in replacements for recurrent PPO architectures, and demonstrate that the simple recurrence and the added degree of freedom from the phase of the complex representations enable significant gains over baselines on several memory-improvable tasks, including continuous control. We further explore how to preserve the phase information of the complex hidden state for a phase-aware policy by drawing a parallel to how quantum states are measured. With our methods reaching up to 2-3 × the reward in environments like rocksample and Craftax compared to the baselines, this work points towards an exciting new direction of representations for RL and the problem of partial observability. Code is available at: https://github.com/Sathya98/qurl
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 57 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
39 of 57 references.
Closest ancestors · 3
Same direction, newest first
- 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains31 Jul
- 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learning2 Sep
- 2024Mastering Memory Tasks with World Models7 Mar
Other references · 36
- 2017Attention Is All You Needcited by 60,361
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
- 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
24 more
- 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
- 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
- 2015Unitary Evolution Recurrent Neural Networkscited by 294
- 2019Stabilizing Transformers for Reinforcement Learningcited by 270
- 2017Deep Complex Networkscited by 217
- 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
- 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
- 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
- 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
- 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
- 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
- 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
- 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
- 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 62
- 2021A Survey of Complex-Valued Neural Networkscited by 61
- 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
- 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 57
- 2022Evaluating Long-Term Memory in 3D Mazescited by 43
- 2017Gated Orthogonal Recurrent Units: On Learning to Forgetcited by 31
- 2022Recurrent networks, hidden states and beliefs in partially observable environmentscited by 12
- 2024Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancycited by 7
- 2023Optimal approximation using complex-valued neural networkscited by 6
- 2015Learning Representations Using Complex-Valued Netscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2018Learning Finite State Representations of Recurrent Policy Networks
- 2026Does Learning Recurrent Networks Matter in Partially Observable RL?
- 2026Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
- 2026Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning
- 2024Recurrent Reinforcement Learning with Memoroids
- 2026Unraveling the Hidden Dynamical Structure in Recurrent Neural Policies
- 2026Hidden States as Value Gradients: The Pontryagin Structure of Recurrent Policies
- 2026Probabilistic Verification of Recurrent Neural Networks for Single and Multi-Agent Reinforcement Learning