mlsanityData to 14 Oct 2026

Improving Controller Generalization with Dimensionless Markov Decision Processes

arXiv 2504.10006 ↗First posted 14 Apr 2025cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Controllers trained with Reinforcement Learning tend to be very specialized and thus generalize poorly when their testing environment differs from their training one. We propose a Model-Based approach to increase generalization where both world model and policy are trained in a dimensionless state-action space. To do so, we introduce the Dimensionless Markov Decision Process (Pi-MDP): an extension of Contextual-MDPs in which state and action spaces are non-dimensionalized with the Buckingham-Pi theorem. This procedure induces policies that are equivariant with respect to changes in the context of the underlying dynamics. We provide a generic framework for this approach and apply it to a model-based policy search algorithm using Gaussian Process models. We demonstrate the applicability of our method on simulated actuated pendulum and cartpole systems, where policies trained on a single environment are robust to shifts in the distribution of the context.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 34 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

21 of 34 references.

References · 21

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  3. 2017Robust Adversarial Reinforcement Learningcited by 451
  4. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  5. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  6. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  7. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 78
  8. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
  9. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  10. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  11. 2018Meta Reinforcement Learning with Latent Variable Gaussian Processescited by 44
  12. 2018Variance reduction properties of the reparameterization trickcited by 41
9 more
  1. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 30
  2. 2023Importance-aware Co-teaching for Offline Model-based Optimizationcited by 28
  3. 2021Multi-Agent MDP Homomorphic Networkscited by 25
  4. 2022Dimensionally Consistent Learning with Buckingham Picited by 22
  5. 2018Learning Robust Optionscited by 17
  6. 2022Dimensionless machine learning: Imposing exact units equivariancecited by 17
  7. 2022Equivariant Networks for Zero-Shot Coordinationcited by 16
  8. 2021Learning Robust Controllers Via Probabilistic Model-Based Policy Searchcited by 1
  9. 2023Using Buckingham's π Theorem for Multi-System Learning Transfer: a Case-study with 3 Vehicles Sharing a Databasecited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Zero-Shot Policy Transfer in Reinforcement Learning using Buckingham's Pi Theoremcited by 1
  2. 2025Direct transfer of optimized controllers to similar systems using dimensionless MPCcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costs
  2. 2025Linear Dynamics meets Linear MDPs: Closed-Form Optimal Policies via Reinforcement Learning
  3. 2025Model-Based Reinforcement Learning Under Confounding
  4. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
  5. 2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts
  6. 2024Model-based Reinforcement Learning for Parameterized Action Spaces
  7. 2024Performative Reinforcement Learning with Linear Markov Decision Process
  8. 2025Addressing Finite-Horizon MDPs via Low-Rank Tensor Value Approximation