mlsanityData to 14 Oct 2026

What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor

arXiv 2609.32532 ↗First posted 26 Sep 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

A generalization gap in reinforcement learning, return on training levels minus return on held-out levels, is usually reported without a reference point. We argue that it should be read against a measured random floor: the return of a uniform-random policy on the same levels under the same evaluation harness. On eight ProcGen environments with PPO at a compute-limited budget (8M steps, 16 parallel environments; three games extended to 25M), the floor changes what standard numbers mean. The test-time action rule decides which policy is measured: in miner, the sampled policy scores 5.1x the floor on held-out levels while its argmax scores below it in every run, and greedy evaluation places two environments significantly below the floor. Used as a convergence diagnostic, raw policy entropy flags six of eight environments, but 32-66% of that entropy lies on actions with identical effects; against the floor, five of eight sampled policies are clearly above it on held-out levels and heist's is not distinguishable from it. An audit of twelve ProcGen codebases finds that nine sample test-time actions with no explicit choice at the evaluation call site. We recommend that every reported gap state its action rule, seed its evaluation and specify its tests before analysis, and report the floor on both level sets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 33 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

30 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale13 Oct
  2. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning29 May

Other references · 28

  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2017Deep Reinforcement Learning that Matterscited by 754
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  7. 2020Reinforcement Learning with Augmented Datacited by 468
  8. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  10. 2020Prioritized Level Replaycited by 198
  11. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  12. 2020Phasic Policy Gradientcited by 123
16 more
  1. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  2. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 87
  3. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  4. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  5. 2023Empirical Design in Reinforcement Learningcited by 55
  6. 2020Evaluating the Performance of Reinforcement Learning Algorithmscited by 36
  7. 2024Simple Policy Optimizationcited by 32
  8. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  9. 2021Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmarkcited by 23
  10. 2022Robust Policy Optimization in Deep Reinforcement Learningcited by 15
  11. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10
  12. 2020Analyzing Reinforcement Learning Benchmarks with Random Weight Guessingcited by 6
  13. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
  14. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  15. 2024Proximal Policy Optimization with Adaptive Explorationcited by 1
  16. 2025PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence
  2. 2026Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO
  3. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning
  4. 2023The Generalization Gap in Offline Reinforcement Learning
  5. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control
  6. 2026Entropy-Preserving Reinforcement Learning
  7. 2025Good Actions Succeed, Bad Actions Generalize: A Case Study on Why RL Generalizes Better
  8. 2024Action abstractions for amortized sampling