# Optimal Transport Meets Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2610.01413 · HTML: https://mlsanity.com/paper/2610.01413/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 28 of its 198 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [The Cramer Distance as a Solution to Biased Wasserstein Gradients](https://mlsanity.com/paper/1705.10743/) (2017)
- [Wasserstein Adversarial Imitation Learning](https://mlsanity.com/paper/1906.08113/) (2019)
- [Wasserstein Robust Reinforcement Learning](https://mlsanity.com/paper/1907.13196/) (2019)
- [Robust Reinforcement Learning with Wasserstein Constraint](https://mlsanity.com/paper/2006.00945/) (2020)
- [On Wasserstein Reinforcement Learning and the Fokker-Planck equation](https://mlsanity.com/paper/1712.07185/) (2017)
- [Improved Complexity Analysis of the Sinkhorn and Greenkhorn Algorithms for Optimal Transport](https://mlsanity.com/paper/2305.14939/) (2023)
- [Risk-Aware Reinforcement Learning through Optimal Transport Theory](https://mlsanity.com/paper/2309.06239/) (2023)
- [Align Your Intents: Offline Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2402.13037/) (2024)
- [A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning](https://mlsanity.com/paper/2404.09946/) (2024)
- [Combining Reinforcement Learning and Optimal Transport for the Traveling Salesman Problem](https://mlsanity.com/paper/2203.00903/) (2022)
- [Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning](https://mlsanity.com/paper/2310.12055/) (2023)
- [Assessing the Impact of Distribution Shift on Reinforcement Learning Performance](https://mlsanity.com/paper/2402.03590/) (2024)
- [Wasserstein Distance Maximizing Intrinsic Control](https://mlsanity.com/paper/2110.15331/) (2021)
- [The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2401.10949/) (2024)
- [Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environments](https://mlsanity.com/paper/2310.08841/) (2023)
- [Optimal Transport-Assisted Risk-Sensitive Q-Learning](https://mlsanity.com/paper/2406.11774/) (2024)
- [Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenters](https://mlsanity.com/paper/2506.15825/) (2025)
- [Safe Wasserstein Constrained Deep Q-Learning](https://mlsanity.com/paper/2002.03016/) (2020)
- [Online Distributionally Robust LLM Alignment via Regression to Relative Reward](https://mlsanity.com/paper/2509.19104/) (2025)
- [Learning Mean-Field Games through Mean-Field Actor-Critic Flow](https://mlsanity.com/paper/2510.12180/) (2025)
- [Equivalence Between Wasserstein and Value-Aware Loss for Model-based Reinforcement Learning](https://mlsanity.com/paper/1806.01265/) (2018)
- [Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport](https://mlsanity.com/paper/2102.07889/) (2021)
- [On Combining Expert Demonstrations in Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2307.10810/) (2023)
- [Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learning](https://mlsanity.com/paper/2502.16328/) (2025)
- [Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport](https://mlsanity.com/paper/2511.19561/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms frequently compare probability distributions, such as state visitation distributions induced by policies and experts, action distributions from learned policies and offline datasets, or transition distributions from learned models and environments. However, commonly used divergences may become ineffective when these distributions overlap weakly, which is frequently encountered in imitation learning, offline RL, and deployment under distribution shift. Optimal transport (OT) offers an alternative by measuring the cost of moving probability mass from one distribution to another under a ground cost that encodes task geometry. This survey covers how OT is used inside RL objectives and algorithms. For each method, we identify: the role OT plays, the distributions compared, the OT formulation used, and the treatment of temporal structure. Beyond categorising existing methods, we discuss the motivations behind different OT choices, practical considerations such as cost design and computational challenges, and highlight open problems including scalable trajectory-level transport, principled handling of mass mismatch, and theoretical analysis for OT-regularised RL.