mlsanityData to 14 Oct 2026

Rethinking Optimal Transport in Offline Reinforcement Learning

arXiv 2410.14069 ↗First posted 17 Oct 2024cs.LGcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose a novel algorithm for offline reinforcement learning using optimal transport. Typically, in offline reinforcement learning, the data is provided by various experts and some of them can be sub-optimal. To extract an efficient policy, it is necessary to stitch the best behaviors from the dataset. To address this problem, we rethink offline reinforcement learning as an optimal transportation problem. And based on this, we present an algorithm that aims to find a policy that maps states to a partial distribution of the best expert actions for each given state. We evaluate the performance of our algorithm on continuous control problems from the D4RL suite and demonstrate improvements over existing methods.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 51 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

37 of 51 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  2. 2023Optimal Transport for Offline Imitation Learning24 Mar
  3. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  4. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  5. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL16 Mar
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2017Improved Training of Wasserstein GANscited by 2,235
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2015Optimal Transport for Domain Adaptationcited by 463
  9. 2016The Option-Critic Architecturecited by 448
  10. 2019Optimal transport mapping via input convex neural networkscited by 206
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  12. 2022Neural Optimal Transportcited by 126
11 more
  1. 2020Primal Wasserstein Imitation Learningcited by 121
  2. 2019Wasserstein-2 Generative Networkscited by 97
  3. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 67
  4. 2022Neural Optimal Transport with General Cost Functionalscited by 36
  5. 2022Kernel Neural Optimal Transportcited by 31
  6. 2020Imitation Learning with Sinkhorn Distancescited by 30
  7. 2022Sliced Optimal Partial Transportcited by 20
  8. 2022Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?cited by 19
  9. 2020Regularized Optimal Transport is Ground Cost Adversarialcited by 17
  10. 2023Extremal Domain Translation with Neural Optimal Transportcited by 12
  11. 2021An Optimal-Transport-Based Reinforcement Learning Approach for Computation Offloadingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transportcited by 6
  2. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  3. 2025Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Mapscited by 1
  4. 2025HOTA: Hamiltonian framework for Optimal Transport Advectioncited by 1
  5. 2025Minimal Ingredients for Reward Assignment from Expert Demonstrationscited by 0
  6. 2026Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learningcited by 0
  7. 2026AnomSeer: Reinforcing Multimodal LLMs to Reason for Time-Series Anomaly Detectioncited by 0
  8. 2026Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transportcited by 0
  9. 2026Test-time Multi-agent Coordination by Decomposed Value Gradient Flowcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimal Transport Meets Reinforcement Learning: A Survey
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  4. 2024To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning
  5. 2024Align Your Intents: Offline Imitation Learning via Optimal Transport
  6. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance
  7. 2022Leveraging Offline Data in Online Reinforcement Learning
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism