# A Taxonomy of Similarity Metrics for Markov Decision Processes

arXiv: https://arxiv.org/abs/2103.04706 · HTML: https://mlsanity.com/paper/2103.04706/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 76 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Scalable methods for computing state similarity in deterministic Markov Decision Processes](https://mlsanity.com/paper/1911.09291/) (2019)
- [Quantifying Differences in Reward Functions](https://mlsanity.com/paper/2006.13900/) (2020)
- [A Principled Approach for Learning Task Similarity in Multitask Learning](https://mlsanity.com/paper/1903.09109/) (2019)
- [REPAINT: Knowledge Transfer in Deep Reinforcement Learning](https://mlsanity.com/paper/2011.11827/) (2020)
- [An Optimal Online Method of Selecting Source Policies for Reinforcement Learning](https://mlsanity.com/paper/1709.08201/) (2017)
- [An Overview of Distance and Similarity Functions for Structured Data](https://mlsanity.com/paper/2002.07420/) (2020)
- [Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfer](https://mlsanity.com/paper/1604.03986/) (2016)
- [Clustering Markov Decision Processes For Continual Transfer](https://mlsanity.com/paper/1311.3959/) (2013)
- [Regret Bounds for Reinforcement Learning with Policy Advice](https://mlsanity.com/paper/1305.1027/) (2013)
- [Self-Organizing Maps as a Storage and Transfer Mechanism in Reinforcement Learning](https://mlsanity.com/paper/1807.07530/) (2018)

## Built on by (most cited first, 6 of 6)

- [Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks](https://mlsanity.com/paper/2111.12064/) (2021)
- [A Framework for Few-Shot Policy Transfer through Observation Mapping and Behavior Cloning](https://mlsanity.com/paper/2310.08836/) (2023)
- [Automaton-Guided Curriculum Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.05271/) (2023)
- [Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Art](https://mlsanity.com/paper/2601.05152/) (2026)
- [A Risk-Averse Preview-based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles](https://mlsanity.com/paper/2112.03232/) (2021)
- [Structural Similarity for Improved Transfer in Reinforcement Learning](https://mlsanity.com/paper/2207.13813/) (2022)

## Abstract (arXiv)

Although the notion of task similarity is potentially interesting in a wide range of areas such as curriculum learning or automated planning, it has mostly been tied to transfer learning. Transfer is based on the idea of reusing the knowledge acquired in the learning of a set of source tasks to a new learning process in a target task, assuming that the target and source tasks are close enough. In recent years, transfer learning has succeeded in making Reinforcement Learning (RL) algorithms more efficient (e.g., by reducing the number of samples needed to achieve the (near-)optimal performance). Transfer in RL is based on the core concept of similarity: whenever the tasks are similar, the transferred knowledge can be reused to solve the target task and significantly improve the learning performance. Therefore, the selection of good metrics to measure these similarities is a critical aspect when building transfer RL algorithms, especially when this knowledge is transferred from simulation to the real world. In the literature, there are many metrics to measure the similarity between MDPs, hence, many definitions of similarity or its complement distance have been considered. In this paper, we propose a categorization of these metrics and analyze the definitions of similarity proposed so far, taking into account such categorization. We also follow this taxonomy to survey the existing literature, as well as suggesting future directions for the construction of new metrics.