# Improving Controller Generalization with Dimensionless Markov Decision Processes

arXiv: https://arxiv.org/abs/2504.10006 · HTML: https://mlsanity.com/paper/2504.10006/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 21 of 21)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [Robust Reinforcement Learning for Continuous Control with Model Misspecification](https://mlsanity.com/paper/1906.07516/) (2019)
- [Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations](https://mlsanity.com/paper/1308.3513/) (2013)
- [RL for Latent MDPs: Regret Guarantees and a Lower Bound](https://mlsanity.com/paper/2102.04939/) (2021)
- [PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos](https://mlsanity.com/paper/1902.01240/) (2019)
- [Meta Reinforcement Learning with Latent Variable Gaussian Processes](https://mlsanity.com/paper/1803.07551/) (2018)
- [Variance reduction properties of the reparameterization trick](https://mlsanity.com/paper/1809.10330/) (2018)
- [SAMBA: Safe Model-Based & Active Reinforcement Learning](https://mlsanity.com/paper/2006.09436/) (2020)
- [Importance-aware Co-teaching for Offline Model-based Optimization](https://mlsanity.com/paper/2309.11600/) (2023)
- [Multi-Agent MDP Homomorphic Networks](https://mlsanity.com/paper/2110.04495/) (2021)
- [Dimensionally Consistent Learning with Buckingham Pi](https://mlsanity.com/paper/2202.04643/) (2022)
- [Learning Robust Options](https://mlsanity.com/paper/1802.03236/) (2018)
- [Dimensionless machine learning: Imposing exact units equivariance](https://mlsanity.com/paper/2204.00887/) (2022)
- [Equivariant Networks for Zero-Shot Coordination](https://mlsanity.com/paper/2210.12124/) (2022)
- [Learning Robust Controllers Via Probabilistic Model-Based Policy Search](https://mlsanity.com/paper/2110.13576/) (2021)
- [Using Buckingham's π Theorem for Multi-System Learning Transfer: a Case-study with 3 Vehicles Sharing a Database](https://mlsanity.com/paper/2310.17545/) (2023)

## Built on by (most cited first, 2 of 2)

- [Zero-Shot Policy Transfer in Reinforcement Learning using Buckingham's Pi Theorem](https://mlsanity.com/paper/2510.08768/) (2025)
- [Direct transfer of optimized controllers to similar systems using dimensionless MPC](https://mlsanity.com/paper/2512.08667/) (2025)

## Abstract (arXiv)

Controllers trained with Reinforcement Learning tend to be very specialized and thus generalize poorly when their testing environment differs from their training one. We propose a Model-Based approach to increase generalization where both world model and policy are trained in a dimensionless state-action space. To do so, we introduce the Dimensionless Markov Decision Process (Pi-MDP): an extension of Contextual-MDPs in which state and action spaces are non-dimensionalized with the Buckingham-Pi theorem. This procedure induces policies that are equivariant with respect to changes in the context of the underlying dynamics. We provide a generic framework for this approach and apply it to a model-based policy search algorithm using Gaussian Process models. We demonstrate the applicability of our method on simulated actuated pendulum and cartpole systems, where policies trained on a single environment are robust to shifts in the distribution of the context.