mlsanityData to 14 Oct 2026

Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale

arXiv 2410.10905 ↗First posted 13 Oct 2024cs.LGcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We demonstrate that recent advances in reinforcement learning (RL) combined with simple architectural changes significantly improves generalization on the ProcGen benchmark. These changes are frame stacking, replacing 2D convolutional layers with 3D convolutional layers, and scaling up the number of convolutional kernels per layer. Experimental results using a single set of hyperparameters across all environments show a 37.9% reduction in the optimality gap compared to the baseline (from 0.58 to 0.36). This performance matches or exceeds current state-of-the-art methods. The proposed changes are largely orthogonal and therefore complementary to the existing approaches for improving generalization in RL, and our results suggest that further exploration in this direction could yield substantial improvements in addressing generalization challenges in deep reinforcement learning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 20 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

18 of 20 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  7. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  8. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  9. 2020Phasic Policy Gradientcited by 123
  10. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  11. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  12. 2019Observational Overfitting in Reinforcement Learningcited by 91
6 more
  1. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  2. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  3. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  4. 2020Rotation, Translation, and Cropping for Zero-Shot Generalizationcited by 24
  5. 2023Explore to Generalize in Zero-Shot RLcited by 20
  6. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2026Maximum Likelihood Reinforcement Learningcited by 36
  2. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  3. 2025Representation Convergence: Mutual Distillation is Secretly a Form of Regularizationcited by 1
  4. 2025AbideGym: Turning Static RL Worlds into Adaptive Challengescited by 1
  5. 2026KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learningcited by 0
  6. 2026What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey Analyzing Generalization in Deep Reinforcement Learning
  2. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
  3. 2026VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
  4. 2025Towards General-Purpose Model-Free Reinforcement Learning
  5. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  6. 2026Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training
  7. 2024GRAM: Generalization in Deep RL with a Robust Adaptation Module
  8. 2025Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?