# Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

arXiv: https://arxiv.org/abs/2006.01096 · HTML: https://mlsanity.com/paper/2006.01096/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 30 of 37)

- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Model-Based Domain Generalization](https://mlsanity.com/paper/2102.11436/) (2021)
- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [Rethinking Domain Generalization for Face Anti-spoofing: Separability and Alignment](https://mlsanity.com/paper/2303.13662/) (2023)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Adversarial Robustness with Semi-Infinite Constrained Learning](https://mlsanity.com/paper/2110.15767/) (2021)
- [What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?](https://mlsanity.com/paper/2305.18882/) (2023)
- [On the Importance of Exploration for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2306.05483/) (2023)
- [The Generalization Gap in Offline Reinforcement Learning](https://mlsanity.com/paper/2312.05742/) (2023)
- [Improving Generalization of Alignment with Human Preferences through Group Invariant Learning](https://mlsanity.com/paper/2310.11971/) (2023)
- [When Is Generalizable Reinforcement Learning Tractable?](https://mlsanity.com/paper/2101.00300/) (2021)
- [Explore to Generalize in Zero-Shot RL](https://mlsanity.com/paper/2306.03072/) (2023)
- [Invariant Policy Learning: A Causal Perspective](https://mlsanity.com/paper/2106.00808/) (2021)
- [Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions](https://mlsanity.com/paper/2205.10218/) (2022)
- [Invariant Causal Imitation Learning for Generalizable Policies](https://mlsanity.com/paper/2311.01489/) (2023)
- [Domain Generalization in Machine Learning Models for Wireless Communications: Concepts, State-of-the-Art, and Open Issues](https://mlsanity.com/paper/2303.08106/) (2023)
- [Evolution and The Knightian Blindspot of Machine Learning](https://mlsanity.com/paper/2501.13075/) (2025)
- [Generalization in Visual Reinforcement Learning with the Reward Sequence Distribution](https://mlsanity.com/paper/2302.09601/) (2023)
- [An Overview of Violence Detection Techniques: Current Challenges and Future Directions](https://mlsanity.com/paper/2209.11680/) (2022)
- [Learning General Policies with Policy Gradient Methods](https://mlsanity.com/paper/2512.19366/) (2025)
- [Learning Causally Invariant Reward Functions from Diverse Demonstrations](https://mlsanity.com/paper/2409.08012/) (2024)
- [Systems with Switching Causal Relations: A Meta-Causal Perspective](https://mlsanity.com/paper/2410.13054/) (2024)
- [Learning Provably Robust Motion Planners Using Funnel Libraries](https://mlsanity.com/paper/2111.08733/) (2021)
- [VIBR: Learning View-Invariant Value Functions for Robust Visual Control](https://mlsanity.com/paper/2306.08537/) (2023)
- [Adaptive Data Exploitation in Deep Reinforcement Learning](https://mlsanity.com/paper/2501.12620/) (2025)
- [Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications](https://mlsanity.com/paper/2512.18135/) (2025)
- [Active Reinforcement Learning over MDPs](https://mlsanity.com/paper/2108.02323/) (2021)
- [Improving generalization to new environments and removing catastrophic forgetting in Reinforcement Learning by using an eco-system of agents](https://mlsanity.com/paper/2204.06550/) (2022)

## Abstract (arXiv)

A fundamental challenge in reinforcement learning is to learn policies that generalize beyond the operating domains experienced during training. In this paper, we approach this challenge through the following invariance principle: an agent must find a representation such that there exists an action-predictor built on top of this representation that is simultaneously optimal across all training domains. Intuitively, the resulting invariant policy enhances generalization by finding causes of successful actions. We propose a novel learning algorithm, Invariant Policy Optimization (IPO), that implements this principle and learns an invariant policy during training. We compare our approach with standard policy gradient methods and demonstrate significant improvements in generalization performance on unseen domains for linear quadratic regulator and grid-world problems, and an example where a robot must learn to open doors with varying physical properties.