mlsanityData to 7 Oct 2026

Mitigating Goal Misgeneralization via Minimax Regret

arXiv 2507.03068 ↗First posted 3 Jul 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Safe generalization in reinforcement learning requires not only that a learned policy acts capably in new situations, but also that it uses its capabilities towards the pursuit of the designer's intended goal. The latter requirement may fail when a proxy goal incentivizes similar behavior to the intended goal within the training environment, but not in novel deployment environments. This creates the risk that policies will behave as if in pursuit of the proxy goal, rather than the intended goal, in deployment -- a phenomenon known as goal misgeneralization. In this paper, we formalize this problem setting in order to theoretically study the possibility of goal misgeneralization under different training objectives. We show that goal misgeneralization is possible under approximate optimization of the maximum expected value (MEV) objective, but not the minimax expected regret (MMER) objective. We then empirically show that the standard MEV-based training method of domain randomization exhibits goal misgeneralization in procedurally-generated grid-world environments, whereas current regret-based unsupervised environment design (UED) methods are more robust to goal misgeneralization (though they don't find MMER policies in all cases). Our findings suggest that minimax expected regret is a promising approach to mitigating goal misgeneralization.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

34 of 41 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  10. 2024Genie: Generative Interactive Environmentscited by 706
  11. 2019Solving Rubik's Cube with a Robot Handcited by 699
  12. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
22 more
  1. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  2. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  3. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  4. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  5. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  6. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  7. 2017Inverse Reward Designcited by 246
  8. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  9. 2020Prioritized Level Replaycited by 190
  10. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  11. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  12. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  13. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  14. 2021Replay-Guided Adversarial Environment Designcited by 108
  15. 2021Consequences of Misaligned AIcited by 79
  16. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  17. 2021Robot Learning from Randomized Simulations: A Reviewcited by 56
  18. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  19. 2023On the Foundation of Distributionally Robust Reinforcement Learningcited by 18
  20. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  21. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  22. 2024Getting By Goal Misgeneralization With a Little Help From a Mentorcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2