mlsanityData to 7 Oct 2026

Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization

arXiv 2205.11104 ↗First posted 23 May 2022cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

At first sight it may seem straightforward to use recurrent layers in Deep Reinforcement Learning algorithms to enable agents to make use of memory in the setting of partially observable environments. Starting from widely used Proximal Policy Optimization (PPO), we highlight vital details that one must get right when adding recurrence to achieve a correct and efficient implementation, namely: properly shaping the neural net's forward pass, arranging the training data, correspondingly selecting hidden states for sequence beginnings and masking paddings for loss computation. We further explore the limitations of recurrent PPO by benchmarking the contributed novel environments Mortar Mayhem and Searing Spotlights that challenge the agent's memory beyond solely capacity and distraction tasks. Remarkably, we can demonstrate a transition to strong generalization in Mortar Mayhem when scaling the number of training seeds, while the agent does not succeed on Searing Spotlights, which seems to be a tough challenge for memory-based agents.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 16 of its 34 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

16 of 34 references.

References · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  8. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  9. 2018Unity: A General Platform for Intelligent Agentscited by 232
  10. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  11. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  12. 2015Memory-based control with recurrent neural networkscited by 77
4 more
  1. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  2. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38
  3. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  4. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 2; 2026: 2.

9 papers.

  1. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  2. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  3. 2025TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learningcited by 13
  4. 2024Towards Safe and Robust Autonomous Vehicle Platooning: A Self-Organizing Cooperative Control Frameworkcited by 1
  5. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learningcited by 1
  6. 2024Decentralized Coordination of Distributed Energy Resources through Local Energy Markets and Deep Reinforcement Learningcited by 0
  7. 2024A Scalable Decentralized Reinforcement Learning Framework for UAV Target Localization Using Recurrent PPOcited by 0
  8. 2025Recurrent Auto-Encoders for Enhanced Deep Reinforcement Learning in Wilderness Search and Rescue Planningcited by 0
  9. 2026Learning to Score: Tuning Cluster Schedulers through Reinforcement Learningcited by 0