A Role of Environmental Complexity on Representation Learning in Deep Reinforcement Learning Agents
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
We developed a simulated environment to train deep reinforcement learning agents on a shortcut usage navigation task, motivated by the Dual Solutions Paradigm test used for human navigators. We manipulated the frequency with which agents were exposed to a shortcut and a navigation cue, to investigate how these factors influence shortcut usage development. We find that all agents rapidly achieve optimal performance in closed shortcut trials once initial learning starts. However, their navigation speed and shortcut usage when it is open happen faster in agents with higher shortcut exposure. Analysis of the agents' artificial neural networks activity revealed that frequent presentation of a cue initially resulted in better encoding of the cue in the activity of individual nodes, compared to agents who encountered the cue less often. However, stronger cue representations were ultimately formed through the use of the cue in the context of navigation planning, rather than simply through exposure. We found that in all agents, spatial representations develop early in training and subsequently stabilize before navigation strategies fully develop, suggesting that having spatially consistent activations is necessary for basic navigation, but insufficient for advanced strategies. Further, using new analysis techniques, we found that the planned trajectory rather than the agent's immediate location is encoded in the agent's networks. Moreover, the encoding is represented at the population rather than the individual node level. These techniques could have broader applications in studying neural activity across populations of neurons or network nodes beyond individual activity patterns.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 25 references that are in the map. First posted on arXiv 3 Jul 2024.
Builds on
11 of 25 references.
References · 11
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
- 2016Learning to Navigate in Complex Environmentscited by 198
- 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
- 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
- 2023Understanding plasticity in neural networkscited by 174
- 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
- 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
- 2023Investigating Navigation Strategies in the Morris Water Maze through Deep Reinforcement Learningcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Embodied World Models Emerge from Navigational Task in Open-Ended Environments
- 2026Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards
- 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning
- 2024A transformer-based deep reinforcement learning approach to spatial navigation in a partially observable Morris Water Maze
- 2026Artifacts as Memory Beyond the Agent Boundary
- 2024Language-Based Augmentation to Address Shortcut Learning in Object Goal Navigation
- 2026From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents
- 2025From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformers