mlsanityData to 7 Oct 2026

State Regularized Policy Optimization on Data with Dynamics Shift

arXiv 2306.03552 ↗First posted 6 Jun 2023cs.LG cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

In many real-world scenarios, Reinforcement Learning (RL) algorithms are trained on data with dynamics shift, i.e., with different underlying environment dynamics. A majority of current methods address such issue by training context encoders to identify environment parameters. Data with dynamics shift are separated according to their environment parameters to train the corresponding policy. However, these methods can be sample inefficient as data are used textitad hoc, and policies trained for one dynamics cannot benefit from data collected in all other environments with different dynamics. In this paper, we find that in many environments with similar structures and different dynamics, optimal policies have similar stationary state distributions. We exploit such property and learn the stationary state distribution from data with dynamics shift for efficient data reuse. Such distribution is used to regularize the policy trained in a new environment, leading to the SRPO (State Regularized Policy Optimization) algorithm. To conduct theoretical analyses, the intuition of similar environment structures is characterized by the notion of homomorphous MDPs. We then demonstrate a lower-bound performance guarantee on policies regularized by the stationary state distribution. In practice, SRPO can be an add-on module to context-based algorithms in both online and offline RL settings. Experimental results show that SRPO can make several context-based algorithms far more data efficient and significantly improve their overall performance.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

43 of 53 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2017Constrained Policy Optimizationcited by 928
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
31 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  2. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  4. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  6. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  9. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  10. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  11. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  12. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  13. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  14. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  15. 2019State Alignment-based Imitation Learningcited by 73
  16. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  17. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  18. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  19. 2016Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Modelcited by 53
  20. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  21. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  22. 2019Environment Probing Interaction Policiescited by 39
  23. 2020State-only Imitation with Transition Dynamics Mismatchcited by 37
  24. 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
  25. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  26. 2019Single Episode Policy Transfer in Reinforcement Learningcited by 17
  27. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  28. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RLcited by 12
  29. 2023Guarded Policy Optimization with Imperfect Online Demonstrationscited by 10
  30. 2020Provably Efficient Model-based Policy Adaptationcited by 8
  31. 2019Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shiftcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 8; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  3. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  4. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 15
  5. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  6. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  7. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  8. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 8
  9. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  10. 2023AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systemscited by 5
  11. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationaritycited by 5
  12. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
11 more
  1. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
  2. 2024Offline Reinforcement Learning with Domain-Unlabeled Datacited by 3
  3. 2024Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learningcited by 2
  4. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  5. 2025FineFT: Efficient and Risk-Aware Ensemble Reinforcement Learning for Futures Tradingcited by 1
  6. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  7. 2025Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learningcited by 0
  8. 2025Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shiftcited by 0
  9. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learningcited by 0
  10. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0
  11. 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  2. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  4. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  5. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation
  6. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
  7. 2023Time-Efficient Reinforcement Learning with Stochastic Stateful Policies
  8. 2024Policy-Guided Diffusion