# Reinforcement Learning in the Wild with Maximum Likelihood-based Model Transfer

arXiv: https://arxiv.org/abs/2302.09273 · HTML: https://mlsanity.com/paper/2302.09273/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 58 references that are in the map.
- First posted on arXiv 18 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [(More) Efficient Reinforcement Learning via Posterior Sampling](https://mlsanity.com/paper/1306.0940/) (2013)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Federated Transfer Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1910.06001/) (2019)
- [Model-based Reinforcement Learning: A Survey](https://mlsanity.com/paper/2006.16712/) (2020)
- [Decoupling Dynamics and Reward for Transfer Learning](https://mlsanity.com/paper/1804.10689/) (2018)
- [Concentration Inequalities for Multinoulli Random Variables](https://mlsanity.com/paper/2001.11595/) (2020)
- [Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration and Planning](https://mlsanity.com/paper/2210.02087/) (2022)
- [Automatic Discovery of Interpretable Planning Strategies](https://mlsanity.com/paper/2005.11730/) (2020)
- [VPE: Variational Policy Embedding for Transfer Reinforcement Learning](https://mlsanity.com/paper/1809.03548/) (2018)
- [Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stability](https://mlsanity.com/paper/2109.11792/) (2021)

## Abstract (arXiv)

In this paper, we study the problem of transferring the available Markov Decision Process (MDP) models to learn and plan efficiently in an unknown but similar MDP. We refer to it as textitModel Transfer Reinforcement Learning (MTRL) problem. First, we formulate MTRL for discrete MDPs and Linear Quadratic Regulators (LQRs) with continuous state actions. Then, we propose a generic two-stage algorithm, MLEMTRL, to address the MTRL problem in discrete and continuous settings. In the first stage, MLEMTRL uses a textitconstrained Maximum Likelihood Estimation (MLE)-based approach to estimate the target MDP model using a set of known MDP models. In the second stage, using the estimated target MDP model, MLEMTRL deploys a model-based planning algorithm appropriate for the MDP class. Theoretically, we prove worst-case regret bounds for MLEMTRL both in realisable and non-realisable settings. We empirically demonstrate that MLEMTRL allows faster learning in new MDPs than learning from scratch and achieves near-optimal performance depending on the similarity of the available MDPs and the target MDP.